Collaborative structured analysis system and method

ABSTRACT

Methods, systems, and apparatus for providing compartmented, collaborative, integrated, automated analytics to analysts are provided. In a first aspect, the present invention provides a computer-implemented method for providing compartmented, collaborative, integrated, automated analytics to analysts including: selecting a computer-encoded project-specific workflow; determining a computer-encoded compartment manager, said computer-encoded compartment manager including computer-encoded information about the context of said project-specific workflow; retrieving said computer-encoded information about the context; selecting a computer-implemented automated analytic using said computer-encoded project-specific workflow; providing under control of said computer-encoded compartment manager said information about the context to said automated analytic; processing said computer-encoded information using said computer-implemented automated analytic, to generate thereby analytical information representing an outcome to said analysts; and processing said analytical information in accordance with said computer-encoded compartment manager and said computer-encoded project-specific workflow.

1 CROSS-REFERENCE TO RELATED U.S. PATENT APPLICATIONS

The present application claims priority under 35 U.S.C. §119(e) toprovisional U.S. Patent Application Ser. No. 61/400,345 filed Jul. 27,2010; and to provisional U.S. Patent Application Ser. No. 61/402,159filed Aug. 25, 2010. Each of the aforementioned patent applications isincorporated herein by reference in its entirety and for all purposes.

2 COPYRIGHT NOTICE

A portion of the disclosure of this patent document may contain materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever. The following notice shall apply to this document:Copyright 2011 Globalytica, LLC.

3 BACKGROUND OF THE INVENTION

3.1 Field of the Invention

The exemplary illustrative technology herein relates to systems,software, and methods for making analytical judgments. It isparticularly useful for issues that require weighing of alternativeexplanations of what has happened, is happening, or is likely to happenin the future. The present invention has applications in the areas ofbusiness and intelligence analysis, criminal forensics, cognitivepsychology, computer science, economics, decision theory, informationprocessing and analysis, and management.

3.2 The Related Art

Analytic activities involve processes to generate hypotheses, to collectand record known relevant information, to categorize relevantinformation as to diagnosticity, reliability, or other factors, to testhypotheses by comparing the hypothesis against relevant information todetermine those hypotheses that are supported by the relevantinformation and those that are not, and to determine and validateindicators for use in acquiring additional relevant information.Analytic activities can be classified as manual, automation assisted,and automated. Manual analytic activities are those that are performedsolely by an analyst, automation assisted analytic activities areperformed by an analyst with automation assistance, and automatedanalytic activities are those activities performed solely by a computer.

Relevant information is information used in analytic activities todetermine which hypotheses are likely, and which are not, or to suggestone or more hypotheses to consider. Relevant information can be physicalevidence, the information gained from analysis of physical evidence,witness reports, photographs, videos, audio recordings, transcripts ofvisual or audio recordings, expert testimony, deductions based on otherrelevant information, computer data, or any other information that canbe used to support one or more hypotheses, to show lack of support forone or more hypotheses, or to suggest one or more possible hypothesis.Where relevant information is absent, but might be expected to bepresent, the lack of relevant information can also constitute relevantinformation. For example, if an aircraft was stolen from an airfield, itwould be expected that the tower records would show a departure by thataircraft around the time it went missing from the airfield. If there isno such departure located, that lack is relevant information in itself,and might support hypotheses that the aircraft was hidden at theairfield rather than stolen, that it was disassembled and removed bytruck, or that it was never present in the first place, while at thesame time reducing support for hypotheses that include the idea of theaircraft being flown away by thieves.

Indicators are observable, or potentially observable, actions,conditions, or events that can be monitored to collect relevantinformation over time. Specific indicators occurring or reachingpre-determined values will support a conclusion that one or morespecific hypotheses has happened, is happening, or is becoming morelikely to happen, while if they do not occur or do not reach thepre-determined values, will support a conclusion that one or morehypotheses did not happen, are not happening, or are less likely tohappen.

Analytic activities typically start by generating a set of hypotheses.The set of hypotheses generated ideally includes all reasonablehypotheses. There are a number of known manual techniques for generatinghypotheses, including, but not limited to, Structured Brainstorming,Nominal Group Technique, the Delphi Method, Multiple HypothesesGeneration and Quadrant Hypothesis Generation. The specific manualhypothesis generation technique selected varies with the training of theanalyst(s), and to some degree, the appropriateness of the technique tothe situation. There are no known examples of automated orautomation-assisted hypothesis generation. Given the number of steps andcalculations involved in carrying out some of the manual techniques, andthe amount of information involved in some steps, the chance foranalysts to make errors is high. Automation of hypothesis generationwould help to reduce the chance for such errors as well as easinganalyst workload and significantly speeding up the analysis processes.

When generating hypotheses, it is necessary to avoid various types ofbias that analysts are prone to which can limit or distort the scope ofthe generated hypotheses and adversely impact the conclusions reached.Some techniques for hypothesis generation have been developed help toavoid some types of bias, but introduce other types of bias. Ways toavoid or limit the effects of bias are needed.

When testing hypotheses as part of analytic activities, it is alsonecessary to avoid various types of bias. Structured analytictechniques, such as Analysis of Competing Hypotheses (ACH), have beendeveloped to reduce some bias effects. Using the ACH technique manuallyis tedious and repetitive, time consuming, does not scale well for largenumbers of hypotheses and relevant information due to the increasingsize of the matrix that results, and does not deal well with a pluralityof analysts since they must either share one matrix and agree onconsistency ratings, or work individually and then manually merge theirconsistency ratings or debate their individual conclusions afterward.When analysts are co-located, the need to share a single matrix, ormanually merge separate results, can also result in “groupthink” bias,as some analysts are improperly influenced in their determinations bythe opinions of other analysts for reasons such as seniority, respect,dislike or other factors.

Each of these techniques can be complex and are slow and awkward toimplement manually without error due to the quantity of informationinvolved and the number of steps and calculations needed. Existingautomation-assisted ACH programs, such as Open Source ACH, address themechanics of the data recording aspects of the technique, and performsome of the calculations required. These programs accentuate biases,such as “anchoring” (i.e. fixating on a first reasonable choice andcomparing subsequent choices to it). They do not support ways to reducebias effects such as anchoring or “groupthink”, do not support thecompartmentalization of information, nor do they support automatedmechanisms for generating hypotheses, do not permit flexible weightingof inputs by analysts (for example, to allow for varying levels ofexperience of the analyst), nor support distinguishing analysts andresults reflecting domain-specific knowledge, and do not support otheraspects of analytic activities, such as identification and evaluation ofindicators, or generation of hypotheses, nor do they provide means totrack analyst progress in rating consistency of hypotheses with relevantinformation, especially when analysts work independently in separatematrices. When performing manual analytic activities, collaborationbetween analysts typically requires that they be co-located, both forcommunication and to have access to the working materials, such as whiteboards, charts, papers, and other means used to record and organizeinformation.

When collaborating during manual analytic activities, it can bedifficult or impossible to maintain compartmentalization of information.Systems and devices to enable easier collaboration between analysts,whether co-located or in diverse locations, while maintaining propercompartmentalization of information, are needed.

Hypotheses or indicators that are common to more than one analytictechnique must be manually copied or entered each time a differentanalytic technique is used to work with them. Doing so with pencil andpaper, or even computerized spreadsheets, is awkward, time consuming andprone to error and does not support shared collaborations andcompartmentalization of information. Analyst notes, assumptions, ordiscussions are not retained or associated with specific information, oreven recorded in the first place, making it difficult or impossible toobtain a complete view of the history of a hypothesis, indicator, oritem of relevant information. Such historical views of these items canprovide insight useful for evaluating the quality of the ultimateconclusions of an analytic project. A means of automating andintegrating the analytic techniques, with automation to reduce theworkload required to implement the individual techniques, that collectsand retains historical information about the origin and handling ofimportant aspects of the analysis, is needed to improve the usability ofthe analysis processes, as well as to increase the quality of results.

Extensible, automated systems are needed for hypothesis generation,hypothesis recording, relevant information recording, hypothesis andrelevant information sharing, hypothesis evaluation, indicator recordingand evaluation, and analytic history recording, all while maintainingrequired compartmentalization of information. Automated andautomation-assisted methods are needed to reduce analyst workload,reduce the likelihood of errors, to assist with identification andrecognition of important relationships, such as which hypotheses a givenpiece of relevant information relates to, which hypotheses areinconsistent with what relevant information, or the reliability of agiven piece of relevant information or its source.

The present invention addresses these and other needs.

4 SUMMARY OF THE INVENTION

The present invention provides methods, systems, and apparatus forproviding compartmented, collaborative, integrated, automated analyticsto analysts. As those having ordinary skill in the art will understandupon reading herein, the methods, systems, apparatus provided by thepresent invention enable extensible, automated systems are needed forhypothesis generation, hypothesis recording, relevant informationrecording, hypothesis and relevant information sharing, hypothesisevaluation, indicator recording and evaluation, and analytic historyrecording, all while maintaining required compartmentalization ofinformation for various purposes.

In a first aspect, the present invention provides a computer-implementedmethod for providing compartmented, collaborative, integrated, automatedanalytics to analysts. In one embodiment, the method provided by theinvention comprises selecting a computer-encoded project-specificworkflow; determining a computer-encoded compartment manager, saidcomputer-encoded compartment manager including computer-encodedinformation about the context of said project-specific workflow;retrieving said computer-encoded information about the context;selecting a computer-implemented automated analytic using saidcomputer-encoded project-specific workflow; providing under control ofsaid computer-encoded compartment manager said information about thecontext to said automated analytic; processing said computer-encodedinformation using said computer-implemented automated analytic, togenerate thereby analytical information representing an outcome to saidanalysts; and processing said analytical information in accordance withsaid computer-encoded compartment manager and said computer-encodedproject-specific workflow. In a more specific embodiment, theproject-specific workflow includes at least one project-specificattribute selected from the group consisting of: guidance to theautomated analytics, as to the process to be followed, information touse as inputs, information required for outputs, and any requiredlabeling, tagging, and compartmentalization. In a still more specificembodiment, guidance to the automated analytics further includesguidance for analysts.

In still another embodiment, the project-specific workflow defines rulesbased upon one or more aspects of the context. In a more specificembodiment, the project-specific workflow defines rules for eachanalyst, each project, for each installation of the system, or by thesystem design.

In another embodiment, the computer-encoded context manager executesunder computer control at least one function selected from the groupconsisting of: generating or assigning tags associated with specificinformation elements, or with specific types of information elementswithin a compartment; generating or assigning compartments associatedwith specific information elements, or with specific types ofinformation elements within a compartment; managing requests to, andinformation elements provided by, a data store to enforce rules forinformation access, tagging, and association rules; assigning orassociating information elements or types of information elements withspecific tags, associations, controls, contexts, or compartments;assigning or associating rules with information elements or types ofinformation elements that require specific tagging or restrictions to beapplied to newly created information elements and restricting theavailability of information elements or types of information elements towhich a requestor is not authorized access or use.

In still another embodiment, the computer-encoded context managerexecutes under computer control at least one function selected from thegroup consisting of: implementing access controls over informationelements; implementing controls over tagging and association amongmultiple information elements; and enforcing information segregation ofinformation elements, including logical and physical segregation ofinformation elements among different data stores.

Yet another embodiment further comprising providing a set of rulesdefining the scope of visibility of information, the rules beingeffective to define private information, restricted information, andunrestricted information.

In another aspect, the present invention provides a computer-implementedsystem for providing compartmented, collaborative, integrated, automatedanalytics to analysts. In one embodiment, the system comprises acomputer-controlled service configured to select a computer-encodedproject-specific workflow; a computer-controlled service configured todetermine a computer-encoded compartment manager, the computer-encodedcompartment manager including computer-encoded information about thecontext of the project-specific workflow; a computer-controlled serviceconfigured to retrieve the computer-encoded information about thecontext; a computer-controlled service configured to selectcomputer-implemented automated analytic using the computer-encodedproject-specific workflow; a computer-controlled service configured toprovide under control of the computer-encoded compartment manager theinformation about the context to the automated analytic; acomputer-controlled service configured to process the computer-encodedinformation using the computer-implemented automated analytic, togenerate thereby analytical information representing an outcome to theanalysts; and a computer-controlled service configured to process theanalytical information in accordance with the computer-encodedcompartment manager and the computer-encoded project-specific workflow.

In a more specific embodiment, the project-specific workflow includes atleast one project-specific attribute selected from the group consistingof: guidance to the automated analytics, as to the process to befollowed, information to use as inputs, information required foroutputs, and any required labeling, tagging, and compartmentalization.In a still more specific embodiment, guidance to the automated analyticsfurther includes guidance for analysts.

In still another embodiment, the project-specific workflow defines rulesbased upon one or more aspects of the context. In a more specificembodiment, the project-specific workflow defines rules for eachanalyst, each project, for each installation of the system, or by thesystem design.

In another embodiment, the computer-encoded context manager executesunder computer control at least one function selected from the groupconsisting of: generating or assigning tags associated with specificinformation elements, or with specific types of information elementswithin a compartment; generating or assigning compartments associatedwith specific information elements, or with specific types ofinformation elements within a compartment; managing requests to, andinformation elements provided by, a data store to enforce rules forinformation access, tagging, and association rules; assigning orassociating information elements or types of information elements withspecific tags, associations, controls, contexts, or compartments;assigning or associating rules with information elements or types ofinformation elements that require specific tagging or restrictions to beapplied to newly created information elements and restricting theavailability of information elements or types of information elements towhich a requestor is not authorized access or use.

In still another embodiment, the computer-encoded context managerexecutes under computer control at least one function selected from thegroup consisting of: implementing access controls over informationelements; implementing controls over tagging and association amongmultiple information elements; and enforcing information segregation ofinformation elements, including logical and physical segregation ofinformation elements among different data stores.

Yet another embodiment further comprising providing a set of rulesdefining the scope of visibility of information, the rules beingeffective to define private information, restricted information, andunrestricted information.

In still another aspect, the present invention provides acomputer-readable medium containing computer-readable program controldevices thereon, the computer-readable program control devices beingconfigured to enable a computer to provide compartmented, collaborative,integrated, automated analytics to analysts by causing the computer toexecute computer-controlled operations comprising: selecting acomputer-encoded project-specific workflow; determining acomputer-encoded compartment manager, said computer-encoded compartmentmanager including computer-encoded information about the context of saidproject-specific workflow; retrieving said computer-encoded informationabout the context; selecting a computer-implemented automated analyticusing said computer-encoded project-specific workflow; providing undercontrol of said computer-encoded compartment manager said informationabout the context to said automated analytic; processing saidcomputer-encoded information using said computer-implemented automatedanalytic, to generate thereby analytical information representing anoutcome to said analysts; and processing said analytical information inaccordance with said computer-encoded compartment manager and saidcomputer-encoded project-specific workflow.

In a more specific embodiment, the project-specific workflow includes atleast one project-specific attribute selected from the group consistingof: guidance to the automated analytics, as to the process to befollowed, information to use as inputs, information required foroutputs, and any required labeling, tagging, and compartmentalization.In a still more specific embodiment, guidance to the automated analyticsfurther includes guidance for analysts.

In still another embodiment, the project-specific workflow defines rulesbased upon one or more aspects of the context. In a more specificembodiment, the project-specific workflow defines rules for eachanalyst, each project, for each installation of the system, or by thesystem design.

In another embodiment, the computer-encoded context manager executesunder computer control at least one function selected from the groupconsisting of: generating or assigning tags associated with specificinformation elements, or with specific types of information elementswithin a compartment; generating or assigning compartments associatedwith specific information elements, or with specific types ofinformation elements within a compartment; managing requests to, andinformation elements provided by, a data store to enforce rules forinformation access, tagging, and association rules; assigning orassociating information elements or types of information elements withspecific tags, associations, controls, contexts, or compartments;assigning or associating rules with information elements or types ofinformation elements that require specific tagging or restrictions to beapplied to newly created information elements and restricting theavailability of information elements or types of information elements towhich a requestor is not authorized access or use.

In still another embodiment, the computer-encoded context managerexecutes under computer control at least one function selected from thegroup consisting of: implementing access controls over informationelements; implementing controls over tagging and association amongmultiple information elements; and enforcing information segregation ofinformation elements, including logical and physical segregation ofinformation elements among different data stores.

Yet another embodiment further comprising providing a set of rulesdefining the scope of visibility of information, the rules beingeffective to define private information, restricted information, andunrestricted information.

The foregoing and still more aspects and advantages of the presentinvention will be made clear when the text herein is read in conjunctionwith the accompanying drawings.

5 BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating the components and the interactionrelationships between various components and automated analytics of anexemplary automated structured analysis system in accordance with oneembodiment of the present invention.

FIG. 2 is a diagram showing some of the information element types andstructure of an exemplary automated structured analysis system'sInformation Store component, in accordance with one embodiment of thepresent invention.

FIG. 3 is a diagram showing how project information elements can befiltered for viewing by analysts, in accordance with one embodiment ofthe present invention.

FIG. 4 is a diagram showing exemplary workflows involving a plurality ofautomated analytics in accordance with an exemplary embodiment of thepresent invention.

FIG. 5 is a flowchart of the steps of the MHG automated analyticcomponent of an exemplary automated structured analysis system, inaccordance with one embodiment of the present invention.

FIG. 6 is a diagram showing functionality supported by the ACH automatedanalytic component of an exemplary automated structured analysis system,in accordance with one embodiment of the present invention.

FIG. 7 is a flowchart of the steps of the QC automated analyticcomponent of an exemplary automated structured analysis system, inaccordance with one embodiment of the present invention.

FIG. 8 is a diagram showing an exemplary 2×2 matrix as used in the QCautomated analytic component of FIG. 7.

FIG. 9 is a flowchart of the steps of the IV automated analyticcomponent of an exemplary automated structured analysis system, inaccordance with one embodiment of the present invention.

FIG. 10 is an exemplary Indicators Validator worksheet and ratingstables.

6 DESCRIPTION OF SOME EMBODIMENTS OF THE INVENTION

Exemplary embodiments of the current invention described herein areintended to illustrate important concepts of the current invention, andto aid those skilled in the art in practicing the invention. They arenot to be considered limiting in any manner on alternative embodimentsthat are not so described.

6.1 Overview

Exemplary embodiments of the present invention implement systems andmethods of making available automated and/or automation-assistedstructured analytic techniques on behalf of, or in collaboration with,optionally distributed sets of users, who are referred to herein as“analysts”. There may be many sets of analysts; each of the sets ofanalysts may overlap or be disjoint with every other set. Analyticactivities supported by the present invention many include automatedversions of manual analytic techniques, automated and/orautomation-assisted association of relevant information with hypothesesand indicators, automated and/or automation-assisted rating, ranking,and/or scoring of hypothesis, indicators, and/or relevant information.Software components that implement one or more aspects of an analyticactivity are called automated analytics. An automated analytic maycompletely automate all of the aspects of an analytic technique,automate portions of an analytic technique, or may provide automatedassistance to analysts in the performance of one or more aspects of theanalytic technique, such as retrieving and organizing information forthe analyst; presenting information to the analyst in defined ways;soliciting, storing, and associating analyst-defined rankings, ratings,associations, and/or comments; or taking other actions such asperforming and/or recording communications between analysts and/ormembers of a set of analysts.

Some exemplary embodiments of the current invention may be implementedas a single automated analytic, as a plurality of automated analytics,and/or in implementations where a combination of features are combinedinto a plurality of automated analytics which may share commoncomponents. In some exemplary embodiments, common components can beimplemented as independent automated analytics, such as an informationmanagement automated analytic, a discussion automated analytic, afiltering automated analytic, etc. as is deemed proper by those skilledin the art. For purposes of description herein, a logical view will beused, where at least some features of the invention will be described asa single monolithic automated analytic, regardless of how they mightactually be embodied in a specific exemplary embodiment.

In particular, aspects of the invention permit sets of analysts,possibly located in different locations or upon different computersystems, to work from a common information base to collaborate andimplement structured analysis techniques while logically and/orphysically separated, while reducing individual analysts' physical andcognitive workload, and providing calculation, management, andinformation compartmentalization of individual and aggregated analystwork products within the context of an analytical project orinvestigation.

Exemplary embodiments of the current invention provideproject-configurable structured automated analytics for generating,testing, and ranking hypothesis and indicators, recording the resultsfrom these automated analytics along with associations with relevantinformation, sharing these items and relevant information, andsupporting and recording collaboration between analysts.

Aspects of the present invention further extend analytic activities bysupporting exchange of information elements between automated analyticsin order to eliminate the need for re-entry of information when movingfrom one automated analytic to another, and so that associatedinformation, such as analyst discussions, assumptions, and other relatedinformation elements and their associations are retained. By providingan automated means to move information elements between automatedanalytics, analyst workload is reduced, the opportunity for errors isreduced, and compartmentalization of information can be maintained.

Exemplary embodiments also comprise automated analytics and rules thatimplement and/or define one or more of the following: filtering ofhypotheses; filtering of relevant information; providing analystdiscussion sessions and associating the results of these sessions withspecific hypotheses, indicators, or other relevant information;capturing analyst assumptions, ratings, and other information providedby analysts, where the outputs of the automated analytics are providedwith additional rules in order to define and/or enforcecompartmentalization of the resulting information; or where theinformation provided to/from an automated analytic is filtered,annotated, or changed in some manner. Examples of these limitations onthe resulting information may include adjustments to previously enteredanalyst inputs or the information that is displayed to the currentanalyst based upon specialized knowledge of individual analysts, thedefined level of visibility of an individual analyst, or the sensitivityof the input information.

In more particular aspects, the systems provided by the invention arecomputer-implemented systems for providing compartmented, collaborative,integrated, automated analytics to analysts. Such embodiments can beimplemented using computers, wherein computer-readable medium containingcomputer-readable program control devices thereon, saidcomputer-readable program control devices being configured to enable acomputer to provide compartmented, collaborative, integrated, automatedanalytics to analysts by causing said computer to executecomputer-controlled operations corresponding to the operations describedherein. The construction operation of such systems and computer-readableprogram media will be familiar to those having ordinary skill in theart.

6.2 Exemplary System Architecture

FIG. 1 illustrates a logical diagram of an exemplary embodiment.Exemplary embodiments of the current invention are implemented usingstandard commercially available computer and network systems(collectively, computing devices), allowing use and access from diverselocations and at diverse times. For clarity, the system is presented asoperating on a single, local computer without limiting the use ofdistributed components and/or distributed computing devices. The currentinvention provides for both local and remote operation of eachcomponent, each connected using well known methods of connections suchas computer networks, message queues, and/or telephony. The specificconnection method used is implementation dependent.

Analysts and Users (1010) interact with the system using the UserInterface component (1110). The User Interface (UI) component (1110)comprises thick- or thin client interfaces to present aspects of theinvention to the users. The UI manages the user interaction, and providemechanisms for the user to authenticate and select one or more projectsthat they will operate under. Once such authorization has beenaccomplished, the UI provides the means for interacting with variouscomponents. Specifically, the UI provides analysts interface with thevarious automated analytics (1150, 1160, 1170, 1175, and 1180) andcommon components on an as-required basis. Specifically, the exemplaryembodiment supports user interface access from a variety of computingdevices, including a standard web browser (e.g. Internet Explorer), aterminal device (e.g. QTERM-G75 from QSI Corporation (Salt Lake City,Utah), a thin client (e.g. X-Windows Server, MS Remote Desktop), or adedicated “thick” client running on a workstation, PC, or other generalpurpose computing device, or on a dedicated hardware or softwareplatform. These will be well understood by those having ordinary skillin the art.

The Project Management component (1120) is configured to defineprojects, including configuring rules for projects, defining sets ofanalysts and their roles in the project, workflows, rules, compartments,project status information, required related information, tags, andassociations, and other actions required to create, configure, andmaintain a project. The Project Management component (1120) recordsproject settings in the Information Store (1250), from which they aremade available to the system, including being made available for use inworkflows and by automated analytics.

The workflow manager (1235) implements project-specific workflows inconjunction with the compartment manager (1130) to make informationavailable and to provide and enforce the project information context toautomated analytics.

The automated analytics (1150, 1160, 1170, 1175, and 1180) provide formanagement of analytic processes, including specifications, informationmanagement, analyst interactions, and the mechanics of the specificanalytic techniques. Automated analytics may make use of various commoncomponents, such as the exemplary common components (described below) ofWeighting (1190), Analyst Discussion (1200), and Annotation (1210). Invarious exemplary embodiments, automated analytics may be embodied asseparate threads, processes or programs. In other exemplary embodiments,they may be combined into a single thread, process, or program. In stillother exemplary embodiments, multiple copies of an automated analyticmay be used. Automated analytics' access to information elements in theinformation store is mediated by the compartment manager component(1130), which can limit or allow the automated analytics to accessinformation elements.

User (1010) interaction with automated analytics (1150, 1160, 1170,1175, and 1180) is mediated by the Compartment Manager component (1130),which can limit or allow an automated analytic to use or display variousinformation elements depending on various combinations of: the user(1010), the information element, how the information element is tagged(or not tagged), the user's role(s), the user's group membership(s), theproject's compartment specifications, the project rules, and otherfactors as described herein. All of the common components' (1190, 1200,& 1210) information access and display are similarly mediated by thecompartment manager.

6.2.1 Compartment Manager

The compartment manager (1130) mediates access to and use of informationelements by the automated analytics in accordance with a compartmentspecification. Compartmentalization includes the steps of identifyingand/or selecting information, identifying and/or selecting theappropriate controls (e.g. access, tag-based, filtering, visibility) toapply to the selected information, restricting access and use of thatinformation in accordance with the selected controls, applying thecontrols to processing activities as appropriate, and applying thecontrols to information generated by the processing activities. Inaddition, the compartment manager mediates subsequent access and use ofinformation elements previously managed as part of a compartment. Acompartment specification is a specification that defines one or moreaspects of the controls required to implement compartmentalization. Thecompartment manager may be implemented as a stand-alone component and/orfully or partially integrated as part of one or more automatedanalytics. The compartment manager has a number of functions, including:

Implementing access controls over information elements.

Implementing controls over tagging and association of informationelements with other information elements.

Enforcing information segregation of information elements, includinglogical and physical segregation of information elements to differentinformation stores.

Application of information segregation includes operations not only forsegregation of information elements, but determining when informationelements and derived information elements are made available toautomated analytics and/or may be displayed to analysts. For example, ananalyst may be permitted to see only “raw” information elements, but notsee the results of other analysts' work (e.g. they may not see thederived information elements). Alternatively, an analyst may bepermitted to only see the results of other analysts' work, but not theunderlying information elements. For example, an automated analytic thatmay use a first set of information elements to calculate a conclusionrepresented as a second set of information element(s), with supportinginformation element(s) associated to this second set of informationelement(s) as defined by rules controlling the first informationelement, where only the resulting conclusion is subsequently madeavailable for viewing by an analyst. These operations may be implementedby combinations of Role-Based Access Control (RBAC), filtering, andother techniques for controlling use and/or availability of informationelements.

The ability to control the creation and use of derived information is aparticularly challenging problem solved by the compartment manager. Inthese cases, automated analytics that use specific information elementsmay:

Have access to the specific information elements restricted, where theelements are not made available to the automated analytic,

Have access to the specific information elements granted, with therequirement that any resulting information be controlled or assigned toa specific compartment, or be tagged in a particular manner,

Have access to the specific information elements granted, wheresubsequent use and/or display of the specific information elements by anautomated analytic are restricted.

For example, a set of information elements is provided to an automatedanalytic for use in a calculation, with the restriction that theunderlying information elements may not be displayed or otherwiseidentified to the user, whilst information elements created as part ofthe calculation are to be assigned compartment controls that permittheir display and identification to analysts.

The compartment manager component performs the following functions inthe system:

Creates, assigns, or removes tags that are associated with specificinformation elements, or with specific types of information elements,within a compartment,

Creates or assigns compartments that are associated with specificinformation elements, or with specific types of information elementswithin a compartment,

Manages requests to, and information elements provided by, aninformation store in order to ensure that specified information access,tagging, and association rules are complied with,

Assigns or associates information elements and/or types of informationelements with specific tags, associations, controls, projects, orcompartments,

Assigns or associates rules with information elements and/or types ofinformation elements that require specific tagging and/or restrictionsto be applied to newly created information elements,

Restricts the availability of information elements and/or types ofinformation elements to which the requestor is not authorized access oruse.

6.2.2 Information Store

Storage for information elements, or any other data used by exemplaryembodiments of the system of the invention, can be implemented using anystorage method or methods known at the time of implementation orinstantiation of elements of the exemplary embodiment, including, forexample, magnetic disk drives, flash memory devices, optical storagedevices, database management systems (DBMSs), network attached storage(NAS) devices.

In some exemplary embodiments, data can be stored on a plurality ofstorage devices. Each storage device can be used to store all or asubset of the data. Such storage can involve duplicating some or alldata and storing a plurality of copies of the data on one or morestorage devices. Duplication of data or choice of storage device can befor reasons of minimizing access delay, maintaining functionality whennetwork communication is impaired or non-existent, to assist withmaintenance of information compartmentalization, or for any otherpurpose determined to be proper by those with skill in the art.

In some exemplary embodiments, regardless of where data is stored, anycomputing device can locally cache any data to which it has access,using any caching method deemed proper by those having skill in the art.FIG. 2 is a diagram representing information elements required by atleast some exemplary embodiments, and the organization of such elementsfor at least some exemplary embodiments. The illustrated informationelements and organization are not limiting on other possible informationelement requirements or arrangements.

An information store (2000) comprises authentication and authorizationdata (2100) that is used to determine whether a given analyst should bepermitted access to the system, and what types of access or connectionsare allowed. This data can comprise user or account names, passwords,multi-factor authentication data, privilege and information accessattributes, location information, system-wide group and ruleinformation, or other information as required and understood by thosehaving skill in the art. In some exemplary embodiments, authenticationand authorization data (2100) can be stored, used, and/or maintained inpart or in whole by the operating system of the hosting device, or bythird party systems. In other exemplary embodiments, authentication andauthorization data (2100) can be stored, used, and/or maintained in partor in whole by the exemplary embodiment of the invention.

Information elements may be stored in an information store. Informationelements comprise:

a hypothesis or set of hypotheses,

an indicator or set of indicators. Indicators are information elementsrepresenting observable, or potentially observable, actions, thresholds,conditions, or events that can be automatically monitored to collectrelevant information over time. Indicators can be assembled intoindicator sets. Indicator sets are useful, for example, for defining aplurality of indicators whose simultaneous or closely timed occurrenceor reaching of predetermined values would suggest that one or morehypotheses about events have occurred, are occurring or are very likelyto occur.

an assumption or a set of assumptions,

an item of relevant information or set of items of relevant information,

a discussion element, or set of discussion elements. Discussion elementscomprise information elements typically collected from the AnalystDiscussion component which may include: entered text, audio recordings,computer transcribed audio, captured e-mails, copies of content,messages, comments made in discussions between team members and relatedmetadata, such as the date and time the comments were made, informationidentifying the analyst that made them, and the context in which theywere made (e.g. the project as a whole, concerning a particularhypothesis, item of relevant information, indicator, assumption, etc.),or other forms of information elements as are determined to be useful bythose with skill in the art.

evaluations and assessments, including, for example consistencyevaluations, validity evaluations, relevance assessments, or credibilityassessments,

information element association information, such as the associationbetween an indicator and a hypothesis or between an assumption and ahypothesis,

tags.

The information store (2000) further comprises one or more analyst datarecords (2200, 2201, 2202, 2203), each of which holds informationrelated to a single analyst. Such information can comprise name andcontact data (2210), status information (2220) such as the projects theanalyst is a member of, experience level, special areas of expertise,etc., and eligibility information (2230), such as whether the analyst ispermitted to be a project lead, security clearance levels, etc.

The information store (2000) also comprises one or more projectinformation records (2300, 2301, & 2302), each of which holdsinformation related to a single project. Such information can comprisedescriptive information about the project (2310), project statusinformation (2320), team data (2330), compartment specifications, rulesand workflows (2335), and information elements, including, by example,hypotheses (2340), relevant information (2350), indicators (2355),assumptions (2360), and other information elements (2390) entered byanalysts, current group matrix information (2370), or discussion data(2380) Additional information, such as message and email queues (notshown) may be integrated into the information store as needs dictate.

Project description information (2310) records hold information such asa project name or ID, an inception date, text describing the issue orissues being investigated, key information sources, etc. Projectdescription information (2310) is entered by the project owner, who isgenerally the project team lead, or a person the project team lead hasdelegated this task to. Project description information is typicallystatic once entered, but can be modified by the project team lead, orteam member assigned to do so, when necessary. Authorization to enter oredit project description information (2310) is specified in exemplaryembodiments by role or by other rule-based specification.

Project status information (2320) records hold information about thecurrent state of the project as a whole, such as whether the project isstill being set up, is active, or has been closed. These records canalso contain additional information the nature of which can depend onthe project state, such as final conclusions reached for a closedproject.

Team data (2330) comprises information about team members; current, pastor anticipated future. This information can refer to analyst datarecords (2200-2203) for each analyst on the team, or comprise additionalinformation, such as the analysts' roles on the project, the start dateof the analysts' participation, end date of the analysts' participation,links to other records related to the analyst (e.g. electronicdiscussion records, assumptions, suggested hypotheses or indicators,etc.).

Compartment specifications, rules and workflows (2335) compriseinformation used to define and enforce compartments, project-specificrules, and to define project-specific workflows. This information can bedefined as a system-wide template and manually or automatically copiedinto each project to be used as-is or as a starting point forproject-specific modifications in some exemplary embodiments. In otherexemplary embodiments, compartment specifications, rules, and workflowsare defined individually for each project.

Hypothesis records (2340) identify and describe hypotheses that theproject is currently using, proposed additional hypotheses that have notyet been accepted for the project, and invalidated hypotheses kept forreference purposes. Additional information, such as the date ahypothesis was entered, it's current acceptance or validity status, theidentity of the analyst who suggested or entered it, links to relatedrelevant information or indicators, etc. can also be included in atleast some exemplary embodiments.

Relevant information records (2350) contain information describingrelevant information that is useful for evaluation of hypotheses. Theyalso include optional associated reason, justification, explanation,and/or ranking information. Relevant information comprises combinationsof content, URL links to sources, date of acquisition, the type of therelevant information (factual, deduced, hearsay, etc.), analystestimates of the reliability of the relevant information, the source ofthe relevant information, or other related information as deemed usefulby those with skill in the art. Relevant information record contentfurther comprises physical evidence, documents, the information gainedfrom analysis of physical evidence, witness reports, photographs,videos, audio recordings, transcripts of visual or audio recordings,expert testimony, deductions based on other relevant information,computer data, or any other information that can be used to support oneor more hypotheses, to show lack of support for one or more hypotheses,or to suggest one or more possible hypotheses. Relevant informationrecords may also comprise computed or calculated values or results sets,such as those determined to be relevant to a hypothesis by an automatedanalytic or other automated process. The computed or calculated valuesmay be part of the content identified by the relevant information recordor may be part of the information providing reason, justification,explanation, and/or ranking information.

Indicator data records (2355) contain information describing indicators.Indicator data records can comprise text, URL links to sources ofrelevant information, pointers to database entries, date of entry, queryor other specifications for computation to perform to assess theindicator, frequency of monitoring, date of last check, analystestimates of the priority of the indicator for assigning monitoringresources, the identity of the analyst or group that suggested theindicator, or other related information as deemed useful by those withskill in the art.

Assumption records (2360) contain information about assumptions enteredby team members. Such information can comprise text descriptions; linksto relevant information, indicators, or hypothesis records that theassumption concerns; links to the analyst record of the team member thatentered the assumption; links to Discussion Data records (2380);estimates of the validity of the assumption entered by various teammembers; or any other information deemed useful by those with skill inthe art.

The group matrix information (2370) comprises the calculation results,references to information elements, and other information elements thatcomprise the displayed information in cells of the group matrixpresentation. The group matrix can, in some alternative embodiments, becalculated as needed, rather than stored in the Information Store(2000).

Discussion Data (2380) comprises information elements or references toinformation elements created by the analyst discussion common component.They represent analyst thinking about the particular subject over time,as well as capture reasoning behind the ratings assigned by the teammembers or other decisions made by an automated analytic.

Other information elements (2390) comprise additional informationelements, such as tagging information, evaluations and assessments, orinformation element associations.

In some exemplary embodiments, each information element is encoded withcompartment specific information at the time of its addition to thesystem.

Information elements are categorized into classes. The classes areimplementation specific and are defined as part of the compartmentrules. Three useful information element classes are base informationelements, derived information elements, and independent informationelements. Base information elements are those information elements thatare input by analysts. Derived information elements are thoseinformation elements that are created by or derived from otherinformation elements. Independent information elements are those thatare created by an automated analytic without reliance upon underlyinginformation elements.

In some exemplary embodiments, information in the information store(2000) is encrypted and is decrypted only for access as the informationis made available to the automated analytic. Encryption and decryptiondecisions are made at the time the information elements are madeavailable to an automated analytic.

6.3 System Elements

In a one aspect, the invention provides a system configured to assistone or more analysts working on a project (collectively referred toherein as a “team”) in breaking a complex analytical problem down intoits component parts, or at least parts having a lesser degree ofcomplexity in comparison to the problem viewed in its entirety: a set ofhypotheses, preferably a set containing a correct hypothesis, relevantinformation and other information elements that are useful in assessingthe set of hypotheses; indicators and other elements that assist inacquiring additional relevant information; and facilities for recordinganalyst assessments regarding the consistency and/or inconsistency ofeach item of relevant information with respect to each hypothesis, orthe diagnosticity of each indicator for each hypothesis, and providingmechanisms for storing this information. The system of the inventionguides one or more analysts (individually, collectively, or in definedgroups) through automated or semi-automated processes that help thempursue their analysis, collect additional relevant information, and/orquestion their assumptions and gain a better understanding of thesubject of the analysis, while integrating each analyst's work productwithin the system in order to provide a unified view of these workproducts for all analysts or across one or more defined groups ofanalysts, while permitting review or monitoring of the analyticalprocess so that the quality of the resulting conclusions can beassessed. In one embodiment, the system provided by the invention isconfigured to assist analysts with automated or semi-automatedgeneration of hypotheses using techniques designed to reduce bias, totest hypotheses, and to identify and evaluate the utility of indicators.In a more particular embodiment, the invention provides automated andsemi-automated processes to:

Compartmentalize information elements.

Effectively visualize one or more information elements from alternativeperspectives,

Extend investigations to find and consider additional relevantinformation or indicators that were not initially considered or known,

Identify, record, and question assumptions,

Identify and document dependencies between hypotheses, relevantinformation, indicators, arguments, assumptions, and comments,

Filter and sort information based upon current perceived relevance,defined filter rules, roles, or group associations, for display or usein computations,

Avoid sources of bias in the generation of hypotheses, the rating ofrelevant information consistency with hypotheses, as well as indetermination of indicators and rating their diagnosticity.

6.3.1 Rules

Other embodiments of the invention also provide for the use of rules tolimit or expand permitted analyst uses of the system, to define howinformation elements are displayed or used in computations, to specifyrequirements, behaviors, default values, and other aspects offunctionality of the system in a manner that permits adjustment offunctionality to meet specific requirements of each embodiment,installation, and/or project.

Rules can be useful in supporting compartmentalization, filtering, androle- or group-based definitions. Rules can specify sets of analysts asindividuals, by role, by group, or by combinations of any of these aswell as defining those excluded from any of the sets. For example, arule can specify applicability to analysts that are members of GroupA,but not members of GroupB, and who have the role of Team Member oralternatively who have the role of Project Owner and who are not Bob orJane.

Rules can specify functional aspects of automated analytics and/oroperations of the system, such as “display information element”,“survey”, “compute value”, or “weight input”, as well as specifyinginformation elements or operations that the specifications apply to,such as “Hypothesis 1”, “items tagged Confidential”, or “computediagnosticity”.

Rules can be used to specify actions (e.g. “add”, “delete”, “view”,“edit”, “rate”), as applied to information elements (e.g. “addhypothesis”, “view relevant information”, “rate indicator”, etc.), withrestrictions based on group membership (e.g. “add hypothesis if in ownergroup”). Rules can also specify restrictions based on tags (e.g. “viewrelevant information if tagged ‘view-by-all’). Rules can also specifycombinations and alternatives, such as “add hypothesis, relevantinformation, or indicator if in owner group or in admin group”, or “viewrelevant information if in GroupA and information tagged‘view-by-GroupA’ or ‘view-by-all’”.

Implementation of rules can be by a variety of techniques wellunderstood by those with skill in the art, such as by use of built-insoftware functionality, dynamically loaded software modules (e.g. DLLsor “plug-ins”), interpreted rules defined by internal mechanisms, orloaded from external sources, such as XML definitions, JSONspecifications, or name/value pair sets. Any combination of thesetechniques can also be used. The specific syntax used to specify rulesis implementation dependent, and will be well understood by those withskill in the art. Some particularly useful types of rules include:

Filtering rules may include aspects such as time of occurrence of arelevant information event, time of collection of an item of relevantinformation, role, group or analyst identity that submitted ahypothesis, item of relevant information, or other input, experiencelevel, group, role or identity of the analyst that entered a hypothesis,item of relevant information, or other input, the source of relevantinformation, or other factors as deemed useful by those with skill inthe art. In some exemplary embodiments, filtering rules can include orsuppress any information elements available to the analyst whosecompartment is in force, based on any attribute of those informationelements as specified by one or more rules.

Aggregation rules define sets of information elements (and associatedinformation elements) to be aggregated, and optionally define anaggregation method for aggregating these defined information elements.Aggregation rules optionally specify creation of one or more informationelements. For example, an aggregation method is a query, a calculation,or other process step defined within an automated analytic that producesa value or values from the set of information elements. Aggregationrules may further specify that aggregated information elements becreated that represents the result of performing an aggregationrule-specified method, or that specific associations between informationelements be made.

Compartment specifications comprise a collection of rules andspecifications, further comprising one or more of ACLs, RBACinformation, information element definitions and specifications,filtering rules, information movement rules, access rules, informationstorage rules, and other rules that affect making information elementsavailable within the system. An example visibility scope rule is anexample of the types of rules defined as part of the compartment rules.One example rule defines three types of information element,analyst-private, restricted-public, or public. Analyst-privateinformation elements are those information elements that are visibleonly to the analyst who entered them; these may be used to recordworking thoughts and assumptions without sharing them with other membersof the team. Restricted-public information elements are thoseinformation elements that are made available to at least one otheranalyst on the team, but are not made generally available as publicinformation. Public information elements include those informationelements which have been made available for sharing to the whole projectteam. In some exemplary embodiments, analyst-private orrestricted-public information elements may require additional publishingand/or approval steps to make them visible to other team members.Similarly, an information storage rule that is part of a compartmentspecification might require all information elements of a specific type(e.g. analyst-private) be stored in a particular information store, ormight require that all information elements created by a specificautomated analytic processing information for a specific compartment betagged with a compartment- or automated analytic-specific tag. In otherexemplary embodiments, a compartment rule might specify that identified(e.g. analyst-private or information elements associated with the“eyes-only” tag) information elements not be shared with other analysts,while still permitting the use of and association with those informationelements in calculations performed by automated analytics.

Weighting rules provide definitions for weighting specific analyst orgroup of analyst results. They can be based on analyst identity, groupmemberships, analyst roles, other analyst and/or group attributes (suchas length of service or whether the analyst or group is identified as asubject matter expert), or a combination of these.

6.3.2 Common Features

Exemplary embodiments comprise features that are common to a pluralityof automated analytics. Common features can be presented differently ineach automated analytic while the feature itself remains common to aplurality of automated analytics. For example, an analyst discussion inan ACH automated analytic can be associated with individual hypotheses,relevant information, or to the group matrix cells where analysts ratethe relevant information with respect to the hypotheses. Analystdiscussions in an IV automated analytic can be associated withindividual indicators, or hypotheses, or to the cells in the group IVmatrix where analysts rate the relevance of indicators to hypotheses.Analyst discussion in a QC automated analytic can be associated withindividual 2×2 matrices, individual quadrants of each generated 2×2matrix, to individual contrary assumptions, or to each of the resultinghypotheses. Despite the differences in association in each automatedanalytic, in each case, the discussion component facilitates thepresentation of hypothesis, indicators, and relevant information to twoor more analysts, and captures discussion results (and possible thediscussion details themselves) that are subsequently associated with thepresented hypothesis, indicators, and relevant information in a mannerthat records the interaction for future review. Variations inpresentation (e.g. a pop-up dialog box, a new window on a screen, or anarea on the automated analytic display dedicated to discussion about aset of currently presented objects, etc.) can be incorporated withoutchanging the basic nature of the feature, or its utility. Commonfeatures are used in one way or another in each of the structuredanalysis automated analytics of exemplary embodiments (MHG, QC, ACH andIV).

Exemplary common features are listed, and then described more fullybelow. Descriptions of exemplary uses in each automated analytic aredescribed in the automated analytic description below. Exemplary commonfeatures comprise:

-   -   Project Management    -   Compartmentalization    -   Group Support    -   Collaborative activities    -   Filtering    -   Analyst Discussion    -   Annotation    -   Tagging    -   Audit Logging

6.3.2.1 Project Management

In some embodiments, project management supports viewing, editing andadding project-related information, such as the project description,project status, compartment specifications, and team data, to theinformation store for a project. This feature is typically notauthorized for use by all analysts, but is restricted to those withappropriate roles.

6.3.2.2 Compartmentalization

For reasons of security, information spread limitation, reduction ofgroup-think, or other purposes, restriction of access to at least someinformation and/or some or all results derived from such information toa subset of analysts working on an analytic project can be beneficial.Such restriction is referred to herein as “compartmentalization”.Compartmentalization of information is useful for a plurality ofreasons, for example maintenance of security, confidentiality ofinformation, evaluation of analysts, or for study of bias effects.Compartmentalization of information and processes is especiallyimportant when working with business and intelligence information, andcomprises a novel aspect of the system.

Aspects of the invention provide mechanisms to supportcompartmentalization of information. In some embodiments, the presentinvention provides for flexible restrictions on access to informationelements. Restricted, or “compartmentalized”, information elements areonly accessible by analysts and/or automated analytics specified by thecompartment specification. Compartmentalization of information elementsis supported in a number of ways in exemplary embodiments.

By permitting analysts to work separately, “groupthink” is reduced oreliminated, and by sharing information through the system, the benefitsof collaboration are preserved, while the mechanics of distributingcurrent ideas and coordinating work flow are taken care of withoutanalyst effort, thus reducing the complexity and opportunity for error.

It is not possible to simply combine known techniques forcompartmentalization with system designs for collaboration andinformation because of the complexity required to integratecompartmentalization rules and controls with the new streams ofinformation generated by collaborative techniques, and to thenselectively enforce the compartmentalization of information whilstmaintaining the collaborative environment across a plurality ofinformation. By enabling compartmentalization as described herein,exemplary embodiments can reduce or eliminate the undesirable spread ofspecific information, without the need to exclude such information fromthe analysis, or some analysts from participation, either of which canlimit the effectiveness of the analysis.

Compartmentalization is supported in a first example where each projectwithin the system of the invention associates all information elementswith information viewing and use restrictions that prohibits access fromunauthorized use within the system. Compartmentalization is supported ina second example embodiment where mechanisms to restrict access to, andor viewing of, specified information elements to specific roles, oranalysts are provided. FIG. 3 is a diagram illustrating this. Theproject data (3000) shown comprises four information elements, Element 1(3010), Element 2 (3020), Element 3 (3030), and Element 4 (3040). Threeanalysts are shown, Analyst 1 (3100), Analyst 2 (3200), and Analyst 3(3300), each of whom has a different view of the project data. Analyst 1(3100) has an unrestricted view that includes all project data(3110-3140). This is a view that would be typical of a project owner.Analyst 2 (3200) has a view that does not include Element 1 (3010), butwhich does include the other three elements from the project (3210, 3220& 3230). Analyst 3 (3300) also has a view that does not include allelements of the project, but Analyst 3's view (3300) is more limitedthan Analyst 2's view (3200), being limited to Elements 2 (3310) and 3(3320). The views shown could result from role-basedcompartmentalization, rule-based compartmentalization, or from acombination of both. Compartmentalization is extended in the presentinvention to encompass the permitted uses of information elements withinautomated analytics, and to actions required upon information elementscreated within those automated analytics. Permitted uses may includespecific actions (e.g. associate with, view, use in calculations) orsets of actions (e.g. opaque use means use in calculations, formassociations with, but not permit viewing by an analyst).

In some more specific embodiments, the compartmentalization is logical,physical, or a combination of both techniques. Logicalcompartmentalization of information is useful when the system of theinvention is hosted upon trusted computing platforms, such that theunderlying operating system and application controls may not besubverted. Logical compartmentalization is enforced through softwaredesign, where the software is designed and implemented to enforce thecompartmentalization. Physical compartmentalization is useful when theunderlying computing platform is not trusted, or when the information isso sensitive that it is deemed important to be able to physically secureportions of the information. Physical compartmentalization is enforcedby physically locating the compartmentalized information in such a waythat it is not accessible to software or users outside of thecompartment, and enforcing the reading and writing of specificinformation elements to the physically separate storage (e.g. a separateinformation store).

Physical compartmentalization of information can involve the use of aplurality of storage devices as described above. Determination ofwhether to duplicate data on a plurality of storage devices, and whichdevice or devices to store particular data on, can be determined atleast in part by system configuration settings or by rules defined forthe system as a whole, or on a project by project basis. For example,one or more rules can be used to define that data tagged in a particularmanner be stored only on storage devices attached to or controlled by aparticular computing device. Rule-based control of data storage cancause data to be stored based, for example, on how data is tagged, thetype of data (hypothesis, relevant information, indicator, annotation,audit log, etc.), the analyst who entered the data, the location orcomputing device where the data was entered, the time or date the datawas entered, what groups have access to the data, or any othercharacteristic deemed relevant or useful by those having skill in theart.

6.3.2.3 Role-Based Access Control

In exemplary embodiments of the invention, the system of the inventionprovides mechanisms to support rules that define role-based accesscontrols for information retrieval, access, viewing/display, and storinginformation aspects of the system.

Implementation of such role-based access control systems is wellunderstood in the art; however, implementation of role-based accesscontrol for structured analysis, as provided by the present invention,is especially important as it is performed as part of thecompartmentalization of information as defined above. One particulardistinction is the use of a tri-part access control; where informationaccess may be blocked, enabled for opaque use, and enabled for visibleuse. Each of these types of access is supported by the role-based accesscontrols of the present invention.

In some embodiments, role-based access controls are integrated with allaspects of the system of the invention. In more specific embodiments,specific information elements are restricted for use and/or access bythose in one or more roles. Related information elements, such asassumptions linked to a particular hypothesis or item of relevantinformation that is itself restricted, are automatically restricted aswell.

Compartment rules can be defined that describe the definition of roles,their functional capabilities assigned to those roles, and the use ofthe role within the compartment. A table of exemplary roles is describedbelow.

Role Example Functional Capabilities Project Owner Assign analysts toroles within the owned project, define analyst groups, assign analyststo groups, define rules for weighting analyst inputs by analyst, group,role, and/or information element or procedure, define rules for displayor access to information elements. Contributor - hypothesis Add a newhypothesis to the suggested hypothesis list for approval by a“Reviewer - hypothesis” role holder. Reviewer - hypothesis Review andaccept a suggested hypothesis into a project. Contributor - relevant Adda new item of relevant information to information the suggested relevantinformation list for approval by a “Reviewer - relevant information”role holder. Contributor - indicator Add a new indicator to thesuggested list of indicators for approval by a “Reviewer - indicator”role holder. Reviewer - relevant Review and accept a suggested item ofinformation relevant information into a project. Reviewer - indicatorReview and accept a suggested indicator into a project. Contributor -analysis Add new analysis information elements (e.g. ratings in an ACHmatrix) to a project. Team Member A person with “Contributor -hypothesis”, “Contributor - relevant information” and/orContributor-indicator”, and “Contributor - analysis” roles.

Role information is configured as needed for each instantiated system,and/or for each project. The roles assigned to individual analysts arespecific to a particular project. A given analyst can be in the role ofTeam Member on a first project, in the role of Project Owner for asecond project, and have no role in a third project.

6.3.2.4 Group-Based Capabilities

For purposes of defining compartmentalization boundaries for informationsharing, weighting of analyst judgments, permitted analyst capabilities,filtering of information displayed or used in calculations, or for otherpurposes it can be useful to be able to specify subsets of analysts as agroup, rather than name them individually, or force them into specificroles. Exemplary embodiments of the invention provide means to definegroups, and to associate analysts with them. Analysts can be members ofa single group, a plurality of groups simultaneously, over time, orboth, or members of no groups. Group names can be used in rules or forother purposes, such as sending e-mail, to refer to all analysts who aremembers of the group.

In some exemplary embodiments, groups are defined relative to a specificproject, and the same group name can be used in disparate projectswithout conflict. Membership in a group in a first project would notprovide membership in a group having the same group name in a secondproject. In other exemplary embodiments, group membership is definedrelative to the system rather than to individual projects, and groupmembership would be the same across all projects in such systems. Forexample, membership in a given group would confer membership in thatgroup in a first project and in a second project. In yet other exemplaryembodiments, project-specific group definitions and system-wide groupdefinitions are both supported, and the scope of a group is defined whena group is created. In some such exemplary embodiments the same groupname cannot be used for both system-wide and project-specific purposes,while in other such exemplary embodiments group names can be used forboth system-wide and project-specific purposes, but only theproject-specific defined group, will be used within the project thatdefines it.

6.3.2.5 Collaborative Activities

A number of common features support collaborative activities of thesystem.

Templates provide reusable and shared definitions for project-basedspecifications. For example, the guidance of the user through thecharacteristic determination process of the MHG automated analytic canbe template-based, and vary from project to project. Templates maydefine one or more aspects of the project. Different analysts may beassigned to use the same template, or to use different templates. Byusing different templates for at least some analysts, unintentional biasresulting from similar questioning patterns is avoided. The decision ofwhich template to use for which project, team, or analyst can be random,be rule-based, be role-based, or any combination of these.

A second common collaboration support aspect of the system is thehandling of groups of analysts (or groups of analysts and automationthat provide results in the same form as analysts). For example, when aset of answers to questions are input by the analysts in response to aprocess of an automated analytic, the analysts selected may only includethose analysts that meet the requirements of specific compartment rules.In some cases, the answers input must be reviewed by one or moreadditional analysts before the answers are made available for use byothers. As with other types of information elements, answers input canbe automatically and/or manually tagged as specified.

When analysts do not work as a complete team to enter answers toquestions or to enter alternatives, the system collects the individualsets of answers and alternatives to enable presentation of a collectivelist of answers and plausible alternatives for each definitionalquestion. Information elements in the collective list can be filtered asrequired to maintain compartmentalization, based on the tags or otherinformation elements associated to each information element. The sharingof questions and analyst inputs, and optionally, the review of thequestions and inputs in a collaborative and filtered environment,materially improves the outcomes of automated analysis processes. Theuse of template-based definitional materials extends the flexibility ofthe analytical technique to a wide range of analytical spaces.

In the case of analysts working in subsets of the team, the system ofthe invention computes a team consensus from the various analyst or teamsubsets that enter answers. Computation of a consensus can involveweighting of inputs based upon weighting rules. Alternatively, analystscan work as a team in this process, reaching a single consensus byagreeable means and inputting the resulting answers using the userinterface.

In exemplary embodiments, providing answers to questions can be dividedand distributed across a project team, and/or may be distributed intime. Selection of analysts to assign work to can be done randomly, byrole, by group memberships, or in combinations of these or othermethods. In some exemplary embodiments, a given question can be assignedto a plurality of analysts or sets of analysts for evaluation ofcredibility. The various answers provided can be averaged, the lowestchosen, the highest chosen, or in some other manner a single answerdetermined for subsequent processing. Division of the work in thismanner extends current manual systems by permitting collaboration andsharing of the workload, and permits decisions to be recorded for futurereview.

6.3.2.6 Filtering

In exemplary embodiments, rule-based control of information display,accesses, and processing enable controlled sharing of informationelements. These limitations on the access and/or use of informationelements are referred to herein as “filtering”.

Aspects of the invention provide mechanisms for filtering the display ofand/or use of information elements and computed results in various wayssuch that chosen subsets of available information or computation resultsare displayed, used in aggregations or otherwise treated in a firstmanner, while other information or computation results are treated in asecond manner. Such filtering is useful in maintainingcompartmentalization of information, to focus on particular aspects ofan analytic project, to consider alternatives, evaluate analysts, andfor study of bias effects. In some exemplary embodiments filtering isachieved through the use of rules in conjunction with roles, groups, andindividual analyst identities, as well as tagging of informationelements, to define how and when filtering is to occur.

Filtering is also used by the compartment manager to determine whichinformation is made available to an automated analytic, and the termsunder which it is made available. For example, the compartment managermay make the determination that a specific piece of information will notbe made available to a specific analyst for the purpose of performing aspecific analytic activity, but may make that same piece of informationavailable to the analyst for a different analytic activity. Similarly,the compartment manager may make the information available to theautomated analytic for use in calculations under the provision that itnot be shown or displayed to the analyst.

By creating groups for subsets of analysts on a team who share aninformation “compartment”, or who have similar expertise ordomain-specific knowledge, and appropriately tagging informationelements, rules can be defined to limit display or use of informationelements to appropriate analysts or to enable weighting of judgments asneeded. For example, analysts can be assigned to project groups such as“cleared-agency-staff”, “consultant”, or “allied-representative”, andhave their access to one or more information elements and/or processes(e.g. rating cells in an ACH matrix) restricted or enabled. Hypothesesthat refer to agency-sensitive materials would be restricted byappropriate rule definitions to members of the “cleared-agency-staff”group, materials that are sensitive would be restricted to members ofthe “cleared-agency staff” or “authorized-consultants” groups, and othermaterials left unrestricted and available to those in any, or no, group.In another example, a consultant (e.g., a forensic expert in a policeinvestigation) can have their access and input restricted to specifichypotheses and information elements within a particular project forwhich they are consulting. In yet another example, a policeinvestigation into a crime that could possibly involve a member of thepolice force, whose identity is not yet known, can optionally restrictcritical relevant information access to the project team lead and thosemembers of the team who are known not to be involved in hopes thatinadvertent revelation of knowledge of that information might help toidentify the suspect, while allowing access to all other relevantinformation to the entire team. This last example points out that evenknowledge of the groups assigned to analysts can require restriction.

The filtering supported by exemplary embodiments of the currentinvention enables team members to compare differences in analyst inputsor values computed from them on an aggregate basis for the entire team,for subsets of the project team (e.g. for various roles or groups ofanalysts), or for one-on-one comparisons between individual analysts.for the ability to view differences for any combination of pairings andgroupings significantly increases the utility of the system for analystsand managers by permitting alternate views of the information to see howindividual analysts compare to each other, or to the team as a whole, todetermine whether there are different “camps” within the team that sharesimilar opinions, or for other reasons. In exemplary embodiments, groupsof analysts may be defined according to various criteria, such asexperience, employing agency, nationality, or other factors orcombinations of factors, such comparisons can involve comparisonsbetween groups possessing such various characteristics.

The information elements available for filtering by a given analyst canbe pre-limited by compartmentalization rules. In some embodiments,computation processes may be restricted to the information available ina particular compartment, while in other embodiments computationprocesses may involve information from other compartments, andcompartmentalization only restricts viewing of the individualinformation elements used in the computation. In the later type ofembodiment, analysts can compare their own or other information incompartments they have access to against computation results thatinclude information elements from other compartments or the entireproject, without having direct access to all of the information used inthe computation. For example, an analyst can answer the question, “Howdoes my evaluation of Hypothesis One compare to the team's?” withoutbeing able to know what ratings other individual team members haveassigned to that hypothesis. This can assist with development of a teamconsensus or enable discussion of such consensus without undue biasbased on member position, reputation, or other factors, and withoutrevealing information restricted by compartmentalization outside of itscompartment.

In some exemplary embodiments, when display of an information element issuppressed by filtering required to maintain compartmentalization, itcan simply be omitted, or be replaced by an alternate display. Thealternate display can, for example, indicate that display of theinformation element is being suppressed, and why. For example,“Hypothesis requires specific group membership for viewing”, or “Item ofrelevant information viewable only by group X members”. In someembodiments, an alternate description specific to the suppressedinformation element can be specified showing to those not possessingmembership in a required group. For example, “hypothesis alpha”, or“Terrorist group planning an attack”, rather than the more specifichypothesis description that would be shown to someone in the requiredgroup, such as “Gray Friday terrorist group planning an attack againstan embassy of the USA in Europe.” In some embodiments a differentalternate description can be provided for each group. In all cases, someindication that the information element is restricted is also present,such as an icon, color coding of the text or cell, etc.

6.3.2.7 Weighting

Aspects of the invention also provide mechanisms to support weighting ofanalyst judgments such that the opinions of some analysts count formore, or less, then those of other analysts in determining the resultsof an operation involving analyst judgment (e.g. when rating relevantinformation vs. hypotheses, or indicators vs. hypotheses). Weighting canbe useful when a particular area of the analysis involvesdomain-specific knowledge and understanding that is possessed by asubset of the analysts on a team. Weighting can also be useful to givemore credence to the opinions of analysts who are experienced with thetechniques being used, and less credence to novices who may notunderstand exactly what is needed. Weighting definitions are provided inrules.

It is common to have a project team with members who possess a range ofexperience levels with structured analysis methods, and who have diversesubject matter expertise. Judgments about relevant information,indicators, or other aspects of an analysis can vary in usefulness basedat least in part on these factors. For instance, if some relevantinformation concerns the activities of a terrorist organization in aparticular country, a team member who has studied that particularterrorist organization extensively may have a different opinion aboutthe relevant information than a team member who only has experience withdiplomatic aspects of that country. It can be reasonable to weigh theterrorist group expert's opinions more strongly than team memberswithout that expertise. To support such needs, exemplary embodiments ofthe current invention can use rules to specify adjustments to analystinputs when using those inputs in calculations such that not all analystinputs have the equal effects on the calculation results. Specificationcan be done using rules that define which analyst or analysts areinvolved, what adjustment is to be made, and which calculations theadjustment is to apply to.

Specification of which analysts' inputs to adjust can be by role, bygroup, by individual analyst identity, or by any combination of these.Rules, as described herein, can be used to define weighting of inputs ina very flexible manner. Specification of the adjustment to be made canbe done with a magnitude, a sign, and a type (e.g. an absolute numberfor inputs that are numbers, percentage change for inputs that arenumbers, or offset step count when the input is chosen from a list), orby other means as will be well understood by those with skill in theart.

6.3.2.8 Analyst Discussion

Exemplary embodiments of the current invention enable analysts toexchange information and opinions in a variety of discussions, such as ageneral discussion associated with a project as a whole, and in morespecific contextual discussions such as discussions associated with eachcell of an ACH group matrix, shared QC matrix, or IV matrix. In someexemplary embodiments, contextual discussions are supported forindividual information elements. Such contextual and general discussionsassists analysts in collaborating and sharing information at many levelsof detail during an analysis, with the discussion threads preserved forfuture reference, whether to ascertain or be reminded of what wasdiscussed, to track the evolution of opinion over time and to determinethe causes of any shifts, or to assess the quality of the conclusionsreached.

In some exemplary embodiments the Analyst Discussion feature isimplemented in whole or in part by means of a project “wiki”. A wiki isa system, generally, but not necessarily, implemented as a website, thatallows the creation and editing of any number of interlinked documents,such as web pages, via a user interface, such as a web browser, and asimplified markup language. Wikis are often used to create collaborativeworks, and generally include a feature to maintain a log of changes,including the time and identity of users making a change. Most can alsomaintain historical versions of all wiki pages for later viewing, andalso generally include the ability to limit access for viewing or formaking additions or changes to the wiki. Such limitations can be tied torule- or role-based compartmentalization features of the invention so asto extend compartmentalization to any wiki incorporated into variousembodiments.

In some other exemplary embodiments of the invention, the AnalystDiscussion feature is implemented using a message system similar to astandard e-mail listserve, where messages are created through the systemof the invention, automatically marked as to the scope of the content(e.g. the project as a whole or a specific information element withinthe project), possibly incorporating links to the project or informationelement for identification and convenience of the receiving analyst, andmade available to other team members. In some exemplary embodiments themessages are made available to all team members. In some other exemplaryembodiments, the messages are sent only to team members who haverequested, or subscribed, to messages related to the particular messagescope. In yet other exemplary embodiments messages are sent only toother team members in roles or groups specified by the sending analyst,or to specific analysts specified by the sending analyst, or to acombination of these. Regardless of how messages recipients aredetermined, the system of the invention retains or receives a copy ofeach message and stores it for future reference. Stored messages aretagged with the tags associated with the scope of the informationelement the messages are associated with, so that thecompartmentalization of the message content and scope are maintained.

In yet other exemplary embodiments the analyst discussion feature isimplemented using video conferencing, voice streaming, SMS or MMSmessaging, e-mail, or any combination of these or other electroniccommunication methods.

In the system of the invention, the Analyst Discussion feature can beused to permit collaborative sharing of information or informationsources, opinions, ideas, images, or other information, and can also beused to assess the quality of conclusions reached by the project team.

6.3.2.9 Annotation of Assumptions

Exemplary embodiments of the current invention provide mechanisms thatallow analysts to document some or all of their assumptions relating toeach information element, and to make such assumptions visible to othermembers of the project team for review and comment, subject only torules used to enforce compartmentalization. In more specificembodiments, for any information element, an analyst can document anyassumptions relating to that specific information element. Assumptionsand annotations are made available subject to compartment restrictionsenforced by the compartment manager.

Knowing what assumptions were made by analysts can be useful forassessing the quality of the conclusions reached. Making documentedassumptions available can be useful for resolving differences of opinionbetween analysts working on an analytic project, and reduce the time toreach a conclusion.

6.3.2.10 Tagging

Information element tagging refers to associating named characteristicswith information elements. The named characteristics (“tags”) can thenbe referenced in rules to cause the rule to be applied to theinformation element so tagged. When an information element is created,whether input by an analyst or computed or generated by the system, tagscan be associated with the information element. In the case ofinformation elements generated or computed by the system, such asanalyst discussion threads or diagnosticity values, information elementsmay be assigned tags, or alternatively may inherit the tags of theinformation elements used in their creation, as defined by the rulesgoverning tagging of created information elements. For example, adiagnosticity value for a given relevant information element may betagged with the tags of the relevant information element. Doing sopreserves the compartmentalization of the source information elementsand prevents unwanted transfer of concepts, facts, and conclusions tothose without access to the source information elements.

Tags may be defined system-wide, on a project-by-project basis, or on acompartment defined basis. In yet other exemplary embodiments, tags aredefined as needed. Project-defined tags are visible only to those whoare members of the defining project and who are permitted to see them,or not prohibited from seeing them, by compartment rules.

Part of the system is the automatic application of tags to informationelements under the direction of the compartment manager (in accordancewith the specific workflow and rules then in effect) when they areaccessed, created, and/or stored. These tags may include informationdescribing the information element, the manner of its creation/use, theanalyst and/or automated analytic using the information element, anyinherited project information, and information elements metadata such assource, date-timestamps, etc.

6.3.2.11 Audit Logging

To enable later review of the progress of an analysis, for analysttraining and development, to assess the quality of the conclusions, topermit restoration of a prior state of the analysis, or for otherpurposes, all analyst inputs, computations, and other activities in thesystem are recorded in an audit log. Audit entries record the time,analyst identity, and the input, computation, or other activityinvolved.

Access to the audit data is restricted in the same manner as access toother project data, so as to maintain compartmentalization ofinformation. In most, but not all, cases the project owner will havefull access to audit data. In some exemplary embodiments, audit data canbe maintained in an encrypted state to limit unauthorized access fromwithin, or from outside of the system of the invention.

6.3.3 Workflows and Project Information

In some exemplary embodiments, the present invention provides for one ormore plug-in automated analytics integrated with project-specificworkflows. The plug-in automated analytic approach permits thefunctionality of the system to be extended in order to supportadditional or differing automated analytics. For example, as additionalautomated analytics are developed, they can be added to the system andthe project-specific workflows adjusted to make the newly addedautomated analytics available to one or more sets of analysts. While afirst exemplary embodiment illustrates implementation of the presentinvention using plug-in techniques, alternate exemplary embodiments maysupport additional automated analytics provided by other approaches,such as embedded automated analytics, cooperating software applicationsoperating in a client-server or peer-to-peer architecture, dynamicallyloaded subroutine libraries, software agents, or other means orcombinations of means that are well understood by those with skill inthe art, without deviating from the scope of the disclosure.

As will be recognized by those skilled in the art, the plug-in automatedanalytic architecture coupled with project-specific workflows supportsdistributed processing, in which one or more automated analytics orother aspects of the system are implemented on distinct processors, withinformation being made available between them.

Some exemplary embodiments use workflows to provide rule-based and/orrole-based guidance for analysts. These workflows may comprisetraditional workflow instructions (steps or sequences of steps to beperformed), automated analytics specifications, analyst guidance,relevant information specifications, information labeling and taggingspecifications, information retrieval, compartment specifications,and/or storage and information routing specifications, authenticationand authorization materials, or analysis task specifications.Collectively, each of these items is part of the project information asdescribed below. Workflows of the present invention are differentiatedfrom traditional workflow systems in that they provide contextualinformation for the operation of the automated analytic and use byanalyst along with the workflow instructions, and thus provide one ormore of the following: guidance to the automated analytics (and theanalysts who use them) as to the process to be followed, information touse as inputs, information required for outputs, and any requiredlabeling, tagging, compartmentalization, or other information requiredfor the analyst to perform their analytic activities using the system.Alternatively, in some exemplary embodiments, a flexible workflowdefines rules based upon one or more aspects of the project, forexample, rules for each analyst, each group, for each installation ofthe system, or by the system design. Workflow rules can be customized onan analyst-by-analyst basis, so that different analysts may havedifferent rules associated with each of their workflows. Thus, ananalyst can be considered a junior analyst within a first workflow andhave a first set of rules defining how the information processed by theanalyst is tagged (e.g. tagged as processed by a junior analyst), whilston a second workflow, the analyst may be a subject matter expert andhave his results tagged in a manner reflective of his status.

In some of these exemplary embodiments, portions of the rules comprisesuggested workflows and/or information routing, tagging, or labelinginstructions. Suggested workflows are helpful to analysts who are notfamiliar with the system and suggest proven patterns of work that arelikely to produce useful results and/or will improve efficiency of theoverall analytic process. In other exemplary embodiments, the portionsof the rules comprise required workflows. In some environments, analyticor business experience, legal requirements, or quality control or otherrequirements of the work dictate that specific approaches to analyticactivities be taken. Forcing (or suggesting) an analyst or group ofanalysts to use a particular workflow is also helpful when analysts arenot co-located and are working independently. In these cases, requiringanalysts to approach the analysis activities in the same manner helpsmaintain information consistency, coordinate activities, keeps analystsactivities synchronized, and facilitates collaboration.

Exemplary embodiments enable connection of the automated analytics suchthat at least some outputs of at least one automated analytic areautomatically made available as inputs to at least one other automatedanalytic. As described above, the inputs and outputs are defined usingaspects of the workflow and may be implemented using communicationsand/or information sharing techniques well known in the art. Notehowever, that the workflow provides each of the automated analyticsproject information required for managing compartmentalization andproject-specific labeling and tagging instructions.

FIG. 4 describes exemplary workflows showing several automated analyticsMHG automated analytic 4100, QC automated analytic 4010, ACH automatedanalytic 4110, IV automated analytic 4310 and automated analytic 4400)and some exemplary, non-limiting, information flows and workflow betweenthem]. Specifically, a first exemplary workflow is shown where the MHGautomated analytic (4100) is used to generate hypotheses that areprovided to the ACH automated analytic (4110) for evaluation in light ofrelevant information, and hypotheses from the ACH automated analytic aremade available to the MHG automated analytic (4100) to be the basis forgenerating additional hypotheses. In like manner, hypotheses are madeavailable between the ACH automated analytic (4110) and the QC automatedanalytic (4010) for generating additional hypotheses for evaluation inthe ACH automated analytic. FIG. 4 also illustrates a second exemplarynon-limiting workflow where the ACH automated analytic (4110) makesavailable hypotheses to a plurality of automated analytics, in thisexample, hypothesis are made available to both the QC automated analytic(4010) and the MHG automated analytic (4100) simultaneously. Each of theQC (4010) and MHG (4100) automated analytics independently generateadditional hypotheses and subsequently make available the additionallygenerated hypothesis to the ACH automated analytic (4110). Additionally,the QC automated analytic (4010) processing generates additionalindicators (4300) which are made available to the IV automated analytic(4310) for evaluation and priority ranking for allocation ofinvestigatory resources. Investigation and/or monitoring of indicatorscan result in additional relevant information (4120), which is madeavailable to the ACH automated analytic (4110) for use in evaluatinghypotheses. Note that the workflow and its associated projectinformation enable each of these components to seamlesslycompartmentalize, interoperate with, and share information, and forinformation generated by each of the automated analytics to beautomatically compartmentalized, labeled, tagged, and associated withother information elements without requiring analyst inputs. Additionalexemplary, non-limiting workflows involving one or more automatedanalytics are possible, and those workflows described herein areprovided as clarifying examples, and should not be viewed as limiting inany way.

An additional exemplary automated analytic is also shown, IndicatorGenerator (4400) that uses automated data mining techniques to searchvarious databases (not shown) in order to determine congruencies betweenhistorical events and thus automatically identify and create additionalindicators, which are then made available to the system. (The IGautomated analytic is not shown as part of the above described exemplaryworkflows.) In this example, the IG automated analytic generatesindicators externally (and asynchronously) from the above exemplarworkflows and makes these indicators available to other automatedanalytics as necessary. This demonstrates that automated analytics neednot be included in a workflow for them to be used as part of a project,and that contextual information may be provided to automated analyticsindependently if they are configured as part of a project. Similarly,the example illustrates that all available automated analytics do nothave to be included in each and every workflow.

Each of the “making available” operations used within a workflow mayimplement one or more storing/retrieving, sharing,transmitting/receiving, and/or transferring steps, in which informationand/or access to the information is made available set or list ofautomated analytics. In one exemplary embodiment, the “making available”operation is controlled and/or accomplished by the exemplary automatedstructured analysis system using project information. Projectinformation comprises copies of, references to, and/or specificationsfor information required by an automated analytic to perform itsfunction. Examples of project information include information storelocation, authentication materials, analyst ID (or a set of analyst IDs)to be used for determination of access to, and display of, information,etc., identifying information for one or more second automated analyticsor other system components that are sharing with or transferringinformation with the first automated analytic, identifying informationfor additional automated analytics that information is to be madeavailable to, identification of the information to be made available (ora copy of it), method of transferring the information elements to andfrom the automated analytic, tagging and labeling of informationelements, as well as any additional information, such as analyst inputs,that may be required or useful to the functioning of the first automatedanalytic and/or its information handling.

In a first exemplary embodiment, project information is at leastinitially created and maintained by a User Interface (UI) component isused to collect data required to create project information, such as theanalyst ID and the project ID the analyst is working on as well as theworkflows, automated analytics, and compartment information to be used.In the first exemplary embodiment, the automated analytics can thenalter the project information in ways dictated by their function so asto cause information flow to and from other automated analytics and/orinformation stores, perform automated tagging and/or labeling, as wellas implement information availability in accordance with compartmentaland workflow requirements.

In a second exemplary embodiment, at last some project information is atleast initially created by a workflow management component. The workflowcomponent is similar to workflow systems well known to those skilled inthe art, with the extended workflow mechanisms to manage and makeavailable project information to the automated analytics. In anotherexemplary embodiment, the workflow system is information driven andinvokes automated analytics in a manner which is governed by informationstored in the project itself. In some exemplary embodiments, the projectinformation is information related to specific outputs from one or moreautomated analytics. For example, a workflow may define a series ofsteps for a process that iterate the steps of: (a) generating andtesting hypothesis, (b) generating and associating indicators withhypothesis, and (c) ranking/scoring hypothesis, until one or morehypothesis are identified as meeting a predefined completion criteria.

One of the challenges overcome by the current invention is the abilityto address the requirements at the intersection of information sharingand information compartmentalization. Information sharing inherentlymakes information available. Information compartmentalization inherentlylimits access to information. One of the challenges addressed by thepresent invention is the ability to both share information and limitinformation sharing in the same system. The problem is made harder ifthe system and/or information is distributed. The workflow system, inwhich project information is managed within the workflow, and isprovided under control of the workflow to each automated analytic,solves this problem by making available to each automated analytic theinformation it requires to access, process, and store information inaccordance with the both the sharing and compartmentalizationrequirements.

6.3.4 Automated Analytics 6.3.4.1 MHG Automated Analytic

Some exemplary embodiments of the current invention comprise automatedanalytic(s) for the generation of hypotheses. One such automatedanalytic is based upon the manual Multiple Hypotheses Generation (MHG)techniques, provided commercially as Multiple Hypothesis Generator™(Pherson Associates, Reston Va.). The MHG automated analytic quicklygenerates large numbers of plausible, mutually exclusive hypotheses, ina manner that is not easily subject to analyst bias, and that cover awide range of possibilities.

The steps comprising MHG automated analytic of the present invention areillustrated in the flow chart of FIG. 5. The MHG automated analyticselects, or has selected for it, compartment-filtered informationelements (generally a hypothesis) or an issue, activity or behavior ofinterest, for use as inputs (5010). An activity or behavior of interestmay be: defined explicitly, either by an information element or userinput, be the outcome of a query, a rule, or be the result of afiltering process applied to the output of other automated analytic(s).The selected input can be an information element, or elements, madeavailable from another automated analytic, such as an ACH automatedanalytic, be acquired from the project information store, or be input byan analyst. In some embodiments, the initial input is derived byselection from among the hypotheses being tested in an ACH automatedanalytic. In some of these exemplary embodiments the selection isautomatically made (e.g. based upon analyst rankings, the hypothesisbest supported by relevant information, a hypothesis randomly chosenfrom among those hypotheses with support above a threshold level, etc.),while in other cases, the selection is performed by an analyst. Thecharacteristics of the input are then determined (5020). Exemplaryembodiments of the current invention also provide automated support fordetermining the characteristics of the inputs. Characteristics may bedetermined by an automated process, a semi-automated process, or anautomation-assisted process such as querying analysts with questionsabout the selected hypothesis, issue, activity, or behavior, todetermine its characteristics and recording their responses. Analystscan be queried as individuals, or as groups. In some exemplaryembodiments, the questions used are built into the system or areconfigured as part of a project, or as part of a template. In anembodiment, exemplary questions are be based on the standardjournalist's questions, “Who, What, Where, When, Why and How?”; however,they can be any other questions determined to be useful by those withskill in the art. Automated forms of questions may be expressed inlanguages appropriate to the automation, such as a query language suchas SQL or one of the XML-based query languages.

Plausible alternatives for each characteristic are then determined(5030). Each alternative characteristic, plausibility assessment, or setof characteristics and assessments may be generated by an analyst orautomatically generated by the MHG using techniques such as lookups ofpreviously known results, evaluation of queries, expert systems, datamining techniques, semantic parsers, rule-based knowledge bases andontologies, and/or a combination of these methods. Input of plausiblealternatives to the characteristics may also be input by analysts.Generated alternatives or alternatives input by analysts can beautomatically and/or manually tagged as specified for the compartment.

All permutations of plausible alternatives are then generated (5040). Itshould be noted that the number of generated permutations can be high,and the steps of determining and recording each permutation and thedetermination of its plausibility, along with all of the requiredcontrols and associated information required to support collaborationand compartmentalization is challenging without the flexible,project-based rules and automation provided by the automated analytics.

Once all permutations of the plausible alternatives are generated, thosepermutations that are illogical or make no sense are discarded (5050).The determination of which permutations to discard is made by analysts,or by use of automated means, such as where permutations match aspecific rule defined for the project, or using automation systems suchas a rule-based expert system. The remaining permutations are then ratedfor credibility in accordance with a project-specific rating scale. Therating process may be conducted using various rating methods, forexample, in parallel with each analyst independently rating the set (orsubset) of the remaining hypothesis, in series, with each analyst ratingsome or all of the set, using the first available analyst, or leastrecently utilized analyst, or in random order. Automated rating methodsusing rules defined within the system are also envisioned.

For example, the score may be assigned using a 0 to 5 point scale, where0 indicates that the permutation makes no sense at all, and values from1 to 5 indicate increasing plausibility. In exemplary embodiments whererating permutations and marking those that do not make sense is aseparate step, the rating scale used might be from 1 to 5 instead. Inyet other alternate embodiments the rating scale can comprise othervalues, such as alphabetic (e.g. A-Z, highest to lowest, colors, realnumbers, or percentages). Credibility rating methods and scores areproject defined and may vary from project to project.

The credibility ratings from a plurality of analysts may be averaged tocalculate a credibility score for each permutation (5060). Thepermutations may be optionally sorted by credibility score (5070).Automated application of rules for data manipulation, including thosefor combining unlike rating schemes (a first set of items are ratednumerically 1-5, and the second set are rated using names), enable theautomated processing of information elements by an automated analytic.

Once those permutations rated for exclusion are made available forfurther processing, the MHG automated analytic optionally filters themfrom the set of permutations made available for subsequent use. In someexemplary embodiments the filtered permutations are discarded. In otherexemplary embodiments the removed permutations are not used in furtherMHG processing. For example, the removed permutations may be displayed(as “grayed out” or otherwise removed from analyst view). In somealternative exemplary embodiments, the rating of permutations that donot make sense is combined with the following step of ratingpermutations for credibility.

In some exemplary embodiments, when a permutation is rated as making nosense, or being below a specific threshold (e.g. a score of 0 meaningthat the combination makes no sense at all), the MHG automated analyticrecords an associated annotation as to the reason. The annotation may bemachine generated or based upon an analyst's response. In some exemplaryembodiments, other permutations that match the reason given are alsoassigned the same rating and reason automatically. For example, if ananalyst indicates that a permutation's “Who” is not capable of doing thepermutation's “What”, then all permutations that include the particular“Who” and “What” are given a credibility of 0 automatically, andannotated with the response indicating that the “Who” is not capable ofdoing the “What”. Similarly, if a given “What” cannot be performed at agiven “Where”, then all permutations comprising that “What” and “Where”get a credibility of 0 and an annotation that the “What” cannot beperformed at the “Where”. This substantially reduces the number ofpermutations that progress to the next level of processing.

The remaining permutations with credibility score above a (possiblydifferent) defined threshold credibility score are selected forconversion into hypotheses (5080). The sorting method and threshold canbe configured for the system. In some cases, threshold is defined by thedesign of the system in some exemplary embodiments. In alternativeexemplary embodiments, the threshold is defined at system installation,or for each project. In yet other exemplary embodiments, the thresholdis calculated automatically using statistical methods, determinationthat ratings are clustered in distinct groupings, with the thresholdselected being between two such clusters, or by other means asdetermined to be proper by those with skill in the art.

Finally, the surviving permutations are restated as hypotheses (5090),and are made available to other processes in the system. In some cases,the converted hypotheses are added to the project information store(5095). In some exemplary embodiments, the conversion of permutationsinto hypotheses is done automatically. In other exemplary embodiments,the conversion is performed under an analyst's guidance and theresulting hypotheses are input into an automated analytic (5095).

After the new hypotheses are made available, the process completes(5100).

In some exemplary embodiments, the system may optionally associateadditional indicators or relevant information when the new hypothesesare created. These additional information elements may be added byauthorized analysts, with review and approval as required, and withoptional tagging to maintain required compartmentalization, or byautomated means such as embodiments that determine additional indicatorsor relevant information using rule-based knowledge bases, expertsystems, ontologies, pattern matching, semantic analysis, orcombinations of these or other techniques well understood by those withskill in the art.

Regardless of how the hypotheses are generated, they must be recorded,associated with other information, such as relevant information, analystratings, analyst comments, etc., be examined for plausibility, bothinitially and as relevant information is acquired, be consideredrelative to each other for likelihood in light of relevant information,and otherwise worked with over the course of an analysis. Exemplaryembodiments that implement MHG automated analytics comprise mechanismsto compartmentalize information and ideas and to enforce thiscompartmentalization while optionally enabling the sharing of resultsand outcomes between analysts, without disclosing compartmentalizedsource information.

6.3.4.2 ACH Automated Analytic

The ACH automated analytic provides automated means for evaluating aplurality of hypotheses against relevant information to determine whichhypotheses are supported by the relevant information and which are not.It incorporates the concept of “diagnosticity” for relevant information,where the more hypotheses a given item of relevant information isconsistent with, the less diagnostic that relevant information elementis.

In one embodiment, the ACH automated analytic process comprises thefollowing steps:

-   -   Identify potential hypotheses. A hypothesis is a testable        proposition about what is true, or about what has happened, is        happening, or will happen. A good hypothesis is worded as a        positive statement that is testable and disprovable, and that is        consistent with all relevant information. A good set of        hypotheses meets two tests. The hypotheses cover all reasonable        possibilities, including those that seem unlikely but not        impossible. And the hypotheses should be mutually exclusive.        That is, if one hypothesis is true, then all other hypotheses        must be false.    -   Arrange relevant information as rows in a matrix, and hypotheses        as columns in the same matrix.    -   In each cell of the matrix, rate how consistent the relevant        information for the row is with the hypothesis of the column.    -   Compute the diagnosticity of each item of relevant information.    -   If there is insufficient relevant information that is        sufficiently diagnostic to reach a conclusion, collect        additional relevant information and repeat the process.        Additional relevant information can be collected by identifying        indicators that are associated with one or more hypotheses.    -   Hypotheses that are inconsistent with relevant information are        discounted. Hypotheses that are most consistent with relevant        information are good candidates for use in MHG or QC automated        analytics to help ensure that all reasonable hypotheses have        been considered.    -   Test conclusions using sensitivity analysis, which weighs how        the conclusion would be affected if key relevant information or        arguments were wrong, misleading, or subject to a different        interpretation. The validity of the most diagnostic relevant        information and the consistency of important arguments are        double-checked to assure that the conclusions' support is sound.    -   Report the lead hypothesis or hypotheses, as well as a summary        of alternatives that were considered, and why they were        rejected. Identify relevant information sources from the process        that can serve as indicators in future analyses.

One aspect of the ACH automated analytic is the generation, use, andmaintenance of an ACH matrix to represent analysts' analysis ofhypotheses with respect to relevant information. An advantage of the ACHautomated analytic is that it scales well with large numbers ofhypotheses and relevant information because automated analysis andfiltering limits the number of hypotheses and the amount of relevantinformation that is made available to an analyst, and that the providedinformation is the most relevant to the current analysis activity.Similarly, the ACH automated analytic supports the use of a plurality ofcompartmented ACH matrices, each filtered for specific uses as definedby their compartments, which reduces analyst workload. The ACH automatedanalytic provides automated merging of results without introduction ofbias. This permits analysts to work independently when necessary, andthen combines their results automatically with automated analyticresults, and provides the resulting matrices of combined results forsubsequent use.

The ACH automated analytic also provides rule-based weighting of inputs,rule-based combination of results, scoring of information elements andsubsequent combination of these scores and results to produce aggregatedscores and results, and the compartment-controlled association withand/or automated evaluation of other information element types (e.g.indicators and assumptions).

The ACH automated analytic of exemplary embodiments containsfunctionality that implements the ACH technique using informationelements entered into the information store, permits entry andalteration of information in the information store, and supports commoncollaborative features of the exemplary embodiments, such as analystdiscussion, filtering and survey processing.

FIG. 6 is a diagram showing the major functions of the ACH automatedanalytic (6000). These are Hypothesis Entry (6020), Relevant informationEntry (6030), Survey Processing (6040), Diagnosticity Calculation(6050), Individual Matrix Processing (6060), Group Matrix Processing(6070), Filtering and Sorting (6080), and Analyst Discussion (6090).

Hypothesis Entry (6020) supports entry and updating of hypothesisrecords associated with a project. In some exemplary embodiments,entering these records is governed by compartment rules. In someexemplary embodiments, analysts who are not authorized by compartmentrules to perform enter and/or update these records may enter suggestedthese records that must be approved an authorized analyst before theybecome visible to or usable by other team members (for example, for usein a group matrix).

Relevant information Entry (6030) enables entry and updating of relevantinformation records for a project. In some exemplary embodiments,entering these records is governed by compartment rules. In someexemplary embodiments, analysts who are not authorized by compartmentrules to enter and/or update these records may enter suggested recordsthat must be approved by an authorized analyst before they becomevisible to or usable by other team members (for example, for use in agroup matrix).

Survey Processing (6040), provided by some exemplary embodiments,provides an automated process that reduces cognitive bias when assigningconsistency ratings to relevant information vs. a particular hypothesis.In some exemplary embodiments, the process involves identifying ananalyst from whom one or more consistency ratings are needed, selectingone or more pairings of a hypothesis and an item of relevant informationfor which consistency ratings are needed from the identified analyst,randomly generating the order in which the ratings of specific pairingsof a hypothesis and relevant information are requested from thatanalyst, and then, using a notification method, requests a rating,optional assumptions, and any other required information for thespecified pairing from the analyst. As will be appreciated by thosehaving ordinary skill in the art, the Survey Processing mechanism ofexemplary embodiments of the current invention can reduce thevulnerability of an analysis to unwanted cognitive bias by presentingthe matrix cells to be rated in random order, so that each analystencounters the decisions differently and with a different mentalcontext.

Individual Matrix Processing (6060) deals with handling of individualanalyst matrices for display of ACH information elements that is withinany compartments that the individual analyst belongs to, as well asentry of the analysts own inputs into the ACH processing (e.g.consistency evaluations, suggested additional hypotheses or relevantinformation, suggested indicators, discussion elements) These matricesare similar to the group matrix, but contain only ratings of a singleanalyst, rather than the team consensus information of the group matrix.

Survey Processing (6040) also supports the evaluation of ACH matrixcells over time by permitting an analyst to begin evaluating an ACHmatrix, end the session before completion, and return at a later time tocomplete more of the ACH matrix until the entire matrix has beenevaluated. Progress indication is displayed in the individual matrixdisplay to show the analyst what percentage of the cells have beenevaluated, and what percentage remains to be evaluated. In someexemplary embodiments, the evaluation status for one or more individualanalysts can be displayed in the group matrix, so that the team'soverall progress can be tracked, and any analysts that are holding upprogress can be identified.

Diagnosticity Calculation (6050) supports calculation of thediagnosticity of one or more relevant information elements with respectto the current set of hypotheses. Diagnosticity is used to sortindividual and group matrices, for example so that the relevantinformation with the highest diagnosticity is located in the top row ofthe group and individual matrices.

Group Matrix Processing (6070) supports handling of the group matrix byconstructing the group matrix and storing it in the information store.The group matrix displays current project hypotheses in columns, andrelevant information in rows. The intersection of a hypothesis columnand a relevant information row is referred to herein as “an ACH cell”,and is used to input and display information elements and/or links toinformation elements concerning the cell (for example, consistencyratings, discussion data, assumptions related to the cell), ion elementsor user interface elements required.

In some exemplary embodiments, the present invention provides a groupmatrix that is configured to display a depiction of collected andcomputed information elements from one or a plurality of analysts,aggregated using an aggregation algorithm that collects a rule-definedset of information elements, typically a set of individual analystassessments, assumptions, and other inputs, and uses these to generate arule-defined consensus view of the assessments, assumptions, and otherinputs from this rule-defined set of analysts, and maps these to cellsin a matrix display that resembles an ACH matrix. In more specificembodiments, the information comprising the group matrix is filtered,sorted, and aggregated in accordance with one or more compartment rules.In yet more specific embodiments, the resulting matrix display presentsthe aggregated results in a form or format consistent ACH matrix. Aswill be understood by those having ordinary skill in the art, such agroup matrix presentation differs from an individual ACH matrixpresentation with respect to the information presented, and theunderlying assumptions and conclusions that can be drawn therefrom. Anindividual ACH matrix presentation includes only those informationelements specified by a particular individual analyst, a group ACHmatrix is a collection of elements aggregated and filtered using one ormore aggregation and filtering rules. This has the effect of enforcingcompartmental segregation of information while enabling distributedprocessing and collaboration between analysts.

In some exemplary embodiments, changes in information elements stored inthe information store will automatically result in the group matrixbeing updated. In other exemplary embodiments, the group matrix isgenerated when needed, and no such updating takes place.

In yet other embodiments, one or more rules that describe the processingof group matrix presentations are provided; these may includeinformation element inclusion/exclusion rules that define theinformation elements eligible for inclusion in the group matrix display,inclusion/exclusion rules based upon analyst identity, groupmemberships, or other criteria, and/or filtering and sortingcalculations, as well as specific calculations for determining one ormore derived values based upon the information provided in one or moreinformation elements.

The group matrix displays information in the context of a team analystfor purposes of maintaining compartmentalization of information. If aspecific analyst identity to use for determining what to display andwhat to exclude has not been specified, the group matrix display islimited by filtering (6080) to showing only those information elementsthat are visible to all team analysts.

In another particular embodiment, the invention provides a cell ratingcalculation, which is configured to combine one or more aspects of theteam members' ratings for the consistency of each item of relevantinformation or indicator with each hypothesis the relevant informationor indicator is relevant to. An example of such a rule might defineratings and their aggregation weighting values as follows:

-   -   CC=2    -   C=1    -   NA=0    -   I=1    -   II=2

Where “CC” means “very consistent” and has an aggregation weightingvalue of two, “C” means “consistent” and has an aggregation weightingvalues of one, “NA” means “not applicable” and has an aggregationweighting values of zero, “I” means “inconsistent” and has anaggregation weighting values of one, and “II” means “very inconsistent”and has an aggregation weighting values of two. The aggregationweighting values associated with the ratings given by each team memberare summed for each of the above categories. The category (CC, C, NA, I,or II) that gets the highest total rating is recorded for that cell inthe group matrix. A tie involving IIs and Is (for example, 2 “II”s and 4“I”s), goes to the “II”. The same is true for ties between “CC” and“C”s. Because “II” and “CC” are relatively rare, it is useful to capturethis situation in the group matrix when it does occur. A tie between“I”s and “C”s goes to the “I”, based on the fact that the methodology isdesigned to identify “I”s rather than “C”s. A tie involving “C” and “NA”defaults to “C” and a tie involving “I” and “NA” defaults to “I”.

In some exemplary embodiments, artificial intelligence techniques, suchas expert systems, rule-based knowledge bases, pattern matching, orothers, can be used to suggest consistency ratings. For example, if ahypothesis suggests that an item was stolen, relevant information thatit was destroyed would be inconsistent, and this type of conclusion canbe determined automatically in at least some cases. Such automatedrating of consistency can speed up the work of rating all cells in anACH matrix, as well as reducing analyst errors when the automaticconsistency rating is used only as a suggestion.

Alternative embodiments can define different ratings and weightingfactors in their rule sets as required.

Filtering (6080) supports inclusion or exclusion of information fromdisplay or processing, such as the Group Matrix Processing (6070), basedon a variety of factors as may be defined in rules specifying thefiltering of information. Filtering may occur in the compartmentmanager, and/or in the presentation of the automated analytic. AnalystDiscussion (6090) supports entry, recording, display, searching,editing, annotating, and reporting of context-dependent discussionsbetween analysts about aspects of a project, such as hypotheses,relevant information, assumptions, the project as a whole, etc. TheAnalyst Discussion component within an ACH automated analytic providesproject information and the user interface specifics needed to associateanalyst messages with specific aspects of ACH processing, such as aparticular hypothesis, ACH matrix cell, or relevant information element,and to associate analyst messages with context-specificcompartmentalization rules.

Collaboratively generated and stored relevant information evaluation,hypothesis suggestion, and discussion results provide additionalopportunities for automated analysis of relevant information orhypotheses, and may detect trends in their evolution over time, and thismay help guide the search for additional relevant information, or beused to assess the quality of the conclusions reached by the projectteam.

The collaborative mechanisms described herein may also provide anopportunity to capture and present historical views of the ACH automatedanalytic, relevant information, hypotheses, and analyst evaluations atparticular times. These historical views may take the form of “point intime” snapshots of the information elements and/or released versions ofparticular analysis results. In either case, the historical views may becaptured using methods such as associating a tag with a particular setof historical information elements and then allowing the group matrix tofilter based at least in part upon such tags.

Some exemplary embodiments of the system of the invention automaticallyanalyze and produce assessments related to one or more aspects ofinformation elements under rule-based controls. For example, the systemcan use the diagnosticity of relevant information and theanalyst-supplied consistency ratings for relevant information tohypotheses to identify the relevant information that is most influentialin judging the available hypotheses and which hypotheses are bestsupported by available relevant information. Analysts can use the systemof the invention to discover where there are differences of opinionwithin the team and, more importantly, whether major differences existregarding the most discriminating relevant information elements, andthereby determine who is disagreeing and employ a collaborationmechanism such as Analyst Discussion, to explore the reasons for theirdifferences. As a remotely usable, multi-user system and method,exemplary embodiments of the current invention reduce the risk ofgroupthink by enabling analysts to work alone initially, providing eachuser with his or her own ACH matrix, and preserve the individualviewpoints independent of the group consensus. Exemplary embodimentsalso enable a group of users to work together when appropriate, with thegroup consensus clearly and flexibly shown in a group ACH matrix andassociated information displays.

6.3.4.3 QC Automated Analytic

The QC automated analytic provides an automated mechanism for generatingadditional hypotheses by challenging key assumptions in currenthypotheses.

FIG. 7 is a flowchart showing the basic steps of the QC automatedanalytic. The first step (7010) is to select a hypothesis from amongthose under consideration. This can be made available by anotherautomated analytic, acquired from the project information store, orinput by an analyst. The hypothesis thought to be most likely at thetime is referred to herein as the “lead” hypothesis. For example, if theproblem is to figure out where the money from a local bank vault went,and given relevant information from witnesses and security camerasshowing three armed robbers taking the money, the lead hypothesis mightbe that three armed robbers entered the bank, threatened customers andstaff with weapons, were given the money, and left with it.

The lead hypothesis is broken down into its component parts and its keyassumptions are identified (7020). The lead hypothesis about the threearmed robbers, would break down into component parts of “three robbers,”“who are armed,” “threatened customers and staff,” “were given themoney,” and “left with the money.” The key assumptions for thesecomponents are that there were exactly three robbers, that they werearmed, that they threatened the customers and staff, that they weregiven all of the missing money, and that they left with the missingmoney.

Once the key assumptions are identified, at least two contraryalternatives are generated for each key assumption (7030). For example,rather than three robbers there might have been a fourth robber outsidethe bank, or there might have been an accomplice inside acting as acustomer, or as a staff member. The robbers might not have been armed .. . the weapons might have been fakes. They might not have threatenedthe customers and staff, but might have offered them a share of themoney if they'd cooperate. Rather than the robbers getting all themissing money, perhaps a cashier hid some of the missing cash inside thebank for later retrieval thinking that the robbers would be blamed forit too, or perhaps the robbers left some cash for the customers andstaff as part of a deal to cooperate. Perhaps rather than leaving, theyjust changed clothes and blended in with the customers, or perhaps theyhid inside the bank somewhere.

Once the pairs of contrary alternatives have been generated, each pairis matched with each other pair and the two pairs are arranged into aseparate 2×2 matrix (7040) such as that shown in FIG. 8. A pair ofcontrary alternatives is referred to in FIG. 8 as either Var A (8020) orVar B (8050)). Var A (8020) and Var B (8050) are arranged in a 2×2matrix (8000) with one pair of contrary alternatives represented by thex axis (8015) and one pair represented by the y axis (8010).

Each pair of contrary alternatives (Var A (8020) or Var B (8050))consists of either two distinct entities or as a two points on a singlecontinuum spectrum. If the two contrary alternatives are points on acontinuum, then the larger or more positive alternative is positioned ateither the top of the y axis (8070) or the right-hand end of the x axis(8040). The smaller or more negative alternative is positioned at eitherthe bottom of the y axis (8060) or the left-hand end of the x axis(8030). The choice of which axis to place the pair on can be madearbitrarily. If the two contrary alternatives making up a pair are notpoints on a continuum, but are simply two distinct alternatives, thepositioning on the chosen axis can be arbitrary. For example, onecontrary alternative pair (Var A (8020)) for a given matrix mightconcern the number of robbers (two more outside (a lookout and a getawaydriver), or an accomplice inside, incognito), and the other contraryalternative pair (Var B (8050)) might be whether they took all the moneyor a cashier hid some for himself). In determining the relativelocations for each pair on their axis, the number of robbers pairrepresents two points on a continuum, with the smaller of the pair beingfour and the larger being five. The five robbers alternative is placedat the right-hand end of the x axis (8040), and the four robbersalternative is placed at the left-hand end of the x-axis (8030). Theother pair (Var B (8050) is placed on the y axis (8010) with the “tookall the money” alternative arbitrarily placed at the high end (8070),and the “a cashier hid some for himself” alternative placed at the otherend (8060). This results in all four possible combinations of the twopairs of contrary alternatives existing in one quadrant or another ofthe 2×2 matrix (8000).

In many cases, though not in all, the upper right quadrant (8080) willcomprise the two most likely contrary alternatives. In our example, thiswould be that there were two additional robbers outside the bank (alookout and a getaway driver) and that the robbers took all the moneywith them. This is a fairly obvious possibility, and shows that theupper right quadrant tends to be fairly boring/predictable.

In many cases, though not in all, the lower left quadrant (8090) has thetwo least likely, or troublesome, contrary alternatives. In our example,this would be that there was an accomplice inside the bank, and that acashier hid some of the money for himself. This is not very likely, butif it were the case, it would be surprising, and in other situations, itmight be the most dangerous possibility due to its unexpectedness. Inmost cases it is worthwhile to consider the upper right and lower leftquadrants first, as they are either the most likely or the mostunexpected possibilities.

The upper left (8100) and lower right (8110) quadrants often containcounter-intuitive combinations, and generally are considered last. Inour example, these would be an accomplice inside the bank, and therobbers taking all the money with them for the upper left quadrant(8100), and there being two more robbers outside and a cashier hidingsome of the money for himself in the lower right quadrant (8110).

Returning to FIG. 7, once the contrary alternatives are arranged in the2×2 matrices (7040), the next step is to create a plausible story foreach quadrant (7050) that combines the contrary alternatives. Forexample, in the current example case, a plausible story could bedeveloped for the lower left quadrant (accomplice inside and a cashierhid some of the money) where the accomplice was a cashier who hid someor all of the money while the other three robbers escaped. At least oneplausible story is needed for each quadrant of each 2×2 2×2 matrix, butadditional stories can be optionally developed and included.

In most cases, resources for investigation are limited, so criteria areselected for deciding which stories are worth investing the resources toinvestigate (7060). Criteria in the current example might includehighest chance of recovering the money, lowest chance of not recognizingall involved criminals, or easiest to verify or rule out. Once thecriteria are chosen (7060), the stories are examined and those meetingthe criteria are selected as deserving the most attention (7070). Theselected stories are then converted to hypotheses (7080), making surethat each meets the criteria for a hypothesis, and stored (7085) in theproject information store. The new hypotheses can optionally be madeavailable to another automated analytic, such as an ACH automatedanalytic (7085).

The next step is to develop indicators (7090) for each new hypothesisand store them (7095) in the project information store. The newindicators can optionally be made available to another automatedanalytic, such as an IV automated analytic (7095). Indicators can beused to collect relevant information that can change the validity of thevarious hypotheses under consideration. If indicators associated with ahypothesis change, the new information provided can change the set ofhypotheses it is deemed worth paying attention to. For example, anindicator for the hypothesis where one of the cashiers was an accompliceof the robbers might be an upward change in spending habits of one ofthe bank's cashiers. If a clerk at the bank suddenly starts spending ata rate inconsistent with past expenditure rates or with known incomelevels, that might indicate that the clerk took some of the missingmoney, or was an inside accomplice of the robbers and was paid off laterand make hypotheses involving either idea more likely.

The indicators are then investigated or monitored to collect relevantinformation that may support or refute one or more hypotheses (7100),which completes the QC process (7110).

QC tends to generate large numbers of hypotheses, each of which hasassociated indicators, but the recording and manipulation of thecontrary assumptions, 2×2 matrices, stories and generated hypotheses aswell as the indicators associated with them can be prohibitive when thetechnique is done manually. Also, QC can suffer from some of the sametypes of biases as ACH. For example, consideration of one 2×2 matrix canresult in a mindset that has effects on the following 2×2 matrixconsiderations. Use of the survey technique when presenting 2×2 matricesto analysts can reduce such effects. Collaboration, with individualanalysts or subsets of the analysts working on a project determiningcontrary alternatives or evaluating 2×2 matrices separately, can reducethe “groupthink” bias effect and result in a wider range of alternativesand stories Support for compartmentalization of information andweighting of judgments during the QC process also enhance the utility ofthe method.

At least some exemplary embodiments of the current invention comprise aQC automated analytic that provides a structured mechanism forgenerating hypotheses, with automated means to reduce analyst workload,maintain compartmentalization of information, support filtering andweighting of inputs and outputs, and record actions for future review oruse in assessing the quality of conclusions.

Exemplary embodiments of the current invention provide automated supportfor input of the selected initial hypothesis. In some embodiments thisinitial input can be derived by selection from among the hypothesesbeing tested in an ACH automated analytic. In some of these exemplaryembodiments the selection is automatic (e.g. the hypothesis bestsupported by relevant information, randomly chosen from among thosehypotheses with support above a threshold level, etc.), while in othersof these exemplary embodiments the selection is performed by an analystusing methods well understood by those skilled in the art of computeruser interface design, such as clicking the item with a mouse, tabbing acursor to the chosen item and pressing a return key, touching an item ona touch screen, or any other means in common use.

Exemplary embodiments also provide additional automated support foranalyst tasks at several levels. The basic level of automated supportautomates such tasks as recording assumptions and contrary assumptionsfor each hypothesis while promoting collaboration between analystswhether co-located or working remotely from each other, generating allpermutations of the pairings of assumptions and contrary assumptionpairs, recording the high/low ratings for each member of each contraryassumption pair, presenting the 2×2 matrices for evaluation to theappropriate analysts to maintain compartmentalization of information inthe order specified by the method configured for the project, recordingthe generated stories for each quadrant of each matrix, and recordingthe indicators developed for each new hypothesis for automatic transferto the IV automated analytic. Additional basic automation includesfunctionality to permit collaborative generation of hypotheses andindicators, recording discussion elements associated with specificmatrices or hypotheses, and presentation of resulting hypotheses andindicators for use by analysts or other software with filtering tomaintain compartmentalization of information, and optional filtering,for example to support comparison of results between analysts, groups,roles, or combinations of these.

In some exemplary embodiments a more advanced level of automation cancomprise a rule processing function combined with a knowledge base, forexample in the form of rules created by subject matter experts (SMEs),along with a natural language processing function in order to assistanalysts with additional QC tasks. The natural language processingfunction can parse hypotheses to determine possible assumptions based onthe sentence structure of an input hypothesis. For example, “Three bankrobbers threatened customers and staff with weapons, took the money andleft with it” could be parsed automatically into “three bank robbers”,“threatened customers and staff with weapons”, and “took the money andleft with it”.

A rule-based knowledge base that contains domain-specific rules canenable automatic generation of contrary assumptions. For example, a rulethat specifies that numbers in assumptions be adjusted up and down couldgenerate “four bank robbers” and “two bank robbers” as contraryassumptions to the “three bank robbers” assumption. Likewise, otherrules might generate contrary assumptions of “pretended to threatencustomers and staff with weapons” and “threatened customers and staffwith fake weapons”, “hid the money and left”, “took only part of themissing money and left”. Such automatically generated contraryassumptions can incorporate learning from many prior events as well asrules generated by methods such as Delphi, crowd-sourcing, etc. Suchautomation can assist less experienced analysts in producing betterresults, and can assist all analysts in avoiding bias in theirconsideration of alternatives. Auto-generation of contrary assumptionsalso reduces analyst workload, time to completion, errors, andresistance to using the technique.

A rule-based knowledge base can also be helpful in the generation ofindicators related to generated hypotheses. By parsing each hypothesisfor key terms, and using these to select relevant rules based on pastevents, SME input, Delphi techniques or other methods, rules can be usedto suggest potentially useful indicators, or be used to automaticallyrate indicators suggested by analysts. Such automation is unlikely to beperfect, and will occasionally generate incorrect results that in somecases will be wildly incorrect, but these situations will be obvious toanalysts, who can simply eliminate the incorrect indicators. Forexample, if the automatic generation of indicators for a bank robberysuggests that the spending habits of bank staff be watched for suddenincreases, analysts will recognize that as a reasonable indicator for ahypothesis involving staff assistance in the robbery, or one involving aclerk helping himself to some of the cash during the robbery. However,if the system generates an indicator that suggests that bank auditors bewatched for spending pattern changes, analysts will recognize it as anerroneous indicator and delete it. Likewise, if the system rates theclerk-watching indicator as not useful, analysts will recognize this asan error when the indicator sorts low during processing.

Analyst input of contrary assumptions is prompted for by the QCautomated analytic. Prompting can be to individual analysts workingalone, or prompting can be to the entire team, or to subsets of theentire team, when analysts are collaborating on contrary assumptioninput. Addition of contrary assumptions to the QC automated analytic insome exemplary embodiments can require that they be input by analystswith a suggestion role or rule-based authority, and in some exemplaryembodiments also require approval by analysts with a reviewer role orrule-based authority to review such inputs before the inputs becomeavailable for use. As with other types of information elements,alternatives input by analysts can be tagged as needed to maintainrequired compartmentalization of information.

When analysts do not work as a complete team to enter contraryassumptions, the system collects the various individual sets of contraryassumptions to enable presentation of a collected team set of contraryassumptions from all analysts. Information elements in the collectiveset can be filtered as required to maintain compartmentalization, basedon the tags assigned to each contrary assumption. The sharing ofcontrary assumptions, and optionally, the review of the contraryassumptions in a collaborative and filtered environment, materiallyimproves the outcomes of the QC automated analytic.

Since exemplary embodiments support the automatic transfer of hypothesesbetween the QC automated analytic and the ACH and MHG automatedanalytics, it is possible to select a hypothesis in the ACH automatedanalytic, use it in the MHG or QC automated analytics as an inputhypothesis, transfer the generated hypotheses back into the ACHautomated analytic for evaluation against known relevant information,select a new lead hypothesis and pass it back through the MHG or QCautomated analytics to generate still more hypotheses. This looping cancontinue until no additional valid hypotheses are being generated, atwhich time it is likely that all useful hypotheses have been generatedand these can then be considered by automated analytics such as the ACHautomated analytics in order to determine the best one in light of therelevant information.

The number of hypotheses generated by the looping approach justdescribed can be quite large. Running the generated hypotheses throughthe ACH automated analytic to sort them by relevant information support,and prioritize them by means such as ease of checking the associatedindicators or the severity of the consequences of failing to considerthem, can be used to direct efforts in the most useful directions, whilenot discarding the harder to evaluate or less well supported hypotheses.The original QC automated analytic did no such prioritizing or sorting.By using exemplary embodiments of the invention as part of the QC-ACHloop, collaboration between analysts is enabled, while avoiding harmfuleffects, such as influence from certain analysts that might affectothers, or biases resulting from the order of consideration ofpossibilities, from adversely affecting the conclusions.

Since the QC automated analytic develops indicators for each hypothesisgenerated, the indicators can remain associated with their hypotheses asthey are fed back into the ACH automated analytic. This can provideassistance in acquiring additional relevant information in the ACHautomated analytic, such as when there is insufficient relevantinformation that is diagnostic, and for evaluating hypotheses in the IVautomated analytic.

Collaboration can also be incorporated into the QC automated analytic insome exemplary embodiments to further reduce the workload of individualanalysts, limit biases, and to stimulate team interaction. When a largenumber of contrary dimensions are being considered, analysts can bedivided into collaborative subsets of the team members and each subsetcan be assigned a different set of 2×2 matrices to work with. Analgorithm can be used for sorting the matrices to maximize the number ofcontrary dimensions each subset is exposed to and must consider.Alternatively, the ACH automated analytic's survey technique can beused, where all analysts review all matrices, but matrices are presentedto each analyst in a unique order, and the results are combined into agroup consensus matrix set for hypothesis generation. Likewise,hypothesis generation can be done by dividing the work between analystsor subsets of team analysts, working as a complete project team, or byanalysts working individually. When done by dividing the work betweenanalysts or groups of analysts, overlap can be incorporated, where aplurality of analysts or subsets process some of the same matrices tomaximize the variety of stories and the resulting hypotheses.

The QC automated analytic can be used to quickly generate large numbersof plausible, mutually exclusive hypotheses, in a manner that is noteasily subject to analyst bias, and that cover a wide range ofpossibilities. By providing automated support to analysts employing theQC automated analytic, and by promoting collaborative use of the method,exemplary embodiments of the current invention reduce analyst workload,reduce the opportunity for errors, maintain compartmentalization ofinformation throughout the QC automated analytic process, and encouragewider deployment of this method of hypotheses generation to enhance thequality of analytic conclusions by enabling the consideration of alarger variety of less biased hypotheses.

6.3.4.4 IV Automated Analytic

Indicators, as described above, can be useful for acquiring relevantinformation for use in ACH processing. Some indicators will provideinformation relevant to a single hypothesis, while other indicators willbe less specific, and will produce information relevant to a pluralityof hypotheses. How specific the relevant information generated frommonitoring an indicator is with respect to a single hypothesis isreferred to as its “diagnosticity”. A high diagnosticity value meansthat relevant information produced by monitoring an indicator isspecific to one, or a very few, potential hypotheses, while a lowdiagnosticity value means that an indicator is associated with many,most, or even all hypotheses being considered. The IV automated analyticprovides a set of automated methods for determining the diagnosticity ofindicators and assisting with a determination of whether additionalindicators are needed for one or more hypotheses. Diagnosticity can be auseful factor in determining an optimal allocation of resources forinvestigation and monitoring of indicators.

When there are a large number of indicators used in an analysis project,there is a need for automated assistance for tracking changes in, oremergence of, indicators over time, determining which indicators producerelevant information and which do not, maintaining the current state ofdiagnosticity for each indicator as hypotheses are added or removed, andmaintaining the relative rankings of indicators for allocation ofinvestigation resources, all while maintaining compartmentalization ofinformation.

FIG. 9 describes the steps used in the IV automated analytic. First, amatrix is generated, where hypotheses under consideration are displayedat the heads of the columns across the top (9010), and indicators aredisplayed down the left side, marking the rows (9020). Indicators aregrouped by the hypothesis they are associated with. For example, ifthere are three hypotheses, A, B, and C, and hypothesis A has threeindicators, and hypothesis B has three indicators, and hypothesis C hastwo indicators, the matrix might appear similar to the one shown in FIG.10 (10000). The three hypotheses are displayed across the top (10010,10020, & 10030) and the indicators are displayed down the left side(10040) as A1, A2, A3, B1, B2, B3, C1, and C2, in that order. For agiven hypothesis, the set of indicators associated with it are known asthe “home indicators”. For hypothesis A, these are A1, A2, and A3(10070). For hypothesis B, these are B1, B2, and B3 (10080). Forhypothesis C, these are C1 and C2 (10090).

Returning to FIG. 9, the next step is to have the analysts rate eachindicator as to consistency with each hypothesis (9030). That is, howlikely the indicator is to appear, change, or take on a particular stateif the given hypothesis has occurred, is occurring, or is about tooccur. For home indicators the ratings will be either “Highly Likely”(HL) or “Likely” (L). If the indicator isn't likely, or highly likely,to indicate the particular hypothesis, it wouldn't be a home indicatorfor the hypothesis. When rating indicators that are not home indicators,such as when rating indicator A1 against hypothesis B in FIG. 10, theratings can be Highly Likely (HL), Likely (L), Could be (C), Unlikely(U), or Highly Unlikely (HU). Each rating is associated with a valuethat varies depending on whether the home indicator in a row is HL or L.FIG. 10 also shows two value tables that hold these ratings (10100 &10200). When the home indicator in a row is HL, the table on the left(10100) provides the values associated with the remaining indicators inthe row. When the home indicator in a row is L, the table on the right(10200) provides the values associated with the remaining indicators.The values for each indicator in a row are added to compute thediagnosticity of the indicator (9040) and these are recorded in thescore column (10060). The higher the total, the higher the diagnosticityof the indicator. The lower the total, the lower the diagnosticity ofthe indicator.

Once the diagnosticity scores have been computed for all indicators, theindicator rows are sorted by diagnosticity, with the most diagnosticindicators are the top (9050). Indicators with low diagnosticity (i.e.they are indicators that will appear, change similarly, and/or take onthe same value for all hypothesis) are eliminated (9060). The remainingindicators are then sorted by hypothesis, and then diagnosticity (9065).If any hypothesis no longer have a sufficient number of indicators withsufficiently high diagnosticity scores, in the opinion of the analysts(9070), additional indicators are determined and added to the matrix(9080) and the process is repeated, otherwise the updated indicatorinformation, such as diagnosticity, the ratings assigned by analysts,etc., is stored (9090) in the project information store and the processis complete (9100).

At least some exemplary embodiments of the current invention comprise anautomated analytic to assist with the validation of indicators using theIV automated analytic, described above. The IV automated analyticprovides a structured mechanism for validating indicators, calculatingtheir diagnosticity, and assisting with sorting indicators for optimaluse of resources for investigating or monitoring them for emergence orchanges in their state. The IV automated analytic provides automatedassistance to reduce analyst workload, maintain compartmentalization ofinformation, support filtering and weighting of inputs and outputs, andrecord actions for future review or use in assessing the quality of theresults.

Exemplary embodiments of the current invention's IV automated analyticprovide automated support for input of indicators generated by otherautomated analytics, such as the QC automated analytic, sorting ofindicators by the hypothesis they were first associated with,construction of the IV matrix with indicators in rows, and hypotheses incolumns, and individual or collaborative input of analyst assessments ofindicator consistency with each hypothesis with automatic calculation ofthe resulting diagnosticity values, sorting of indicators bydiagnosticity, inclusion of incorporation of rule-based weightingfactors, while maintaining compartmentalization of information.

In some exemplary embodiments, artificial intelligence techniques, suchas expert systems, rule-based knowledge bases, pattern matching, orothers, can be used to suggest consistency ratings. For example, if ahypothesis deals with movement of shipping containers by rail, anindicator based on weather at sea would be inconsistent, and this typeof conclusion can be determined automatically in at least some cases.Such automated rating of consistency can speed up the work of rating allcells in an IV matrix, as well as reducing analyst errors when theautomatic consistency rating is used only as a suggestion.

In some exemplary embodiments, indicators with diagnosticity valuesbelow a specified threshold value are displayed differently from thoseabove the threshold, and are not considered for monitoring orinvestigation. Such indicators are retained however, both for historicaltracking and because changes in the hypotheses being considered, or inanalyst assessments of the consistency of an indicator with a hypothesiscan alter the diagnosticity of the indicator and possibly move it abovethe threshold value.

In other exemplary embodiments, indicators have their diagnosticityvalues examined automatically to determine if they are “clustered” . . .that is, they are in distinct groups where the indicators making up agroup have intra-group diagnosticity values that differ by a smallamount compared to inter-group diagnosticity value differences. If theindicators are clustered into two distinct groups, the group with thehigher diagnosticity values is retained as useful, and the group withthe lower diagnosticity values is not considered for monitoring orinvestigation. If there are not two distinct groups the thresholdtechnique described above can be used to determine which indicators areuseful.

As indicators produce relevant information, and this information isadded to a project's information store, audit logging will record theaddition of the relevant information. At least some exemplaryembodiments also record information as to which indicator or indicatorsproduced the relevant information, and to determine which indicators aremost productive of relevant information. The results of suchdeterminations can be used to determine specific indicators to suggestin future analytic projects, or as additional input into rating ofindicators for determining allocation of investigatory resources.

7 IMPLEMENTATION

The invention can be implemented in digital electronic circuitry, or incomputer hardware, firmware, software, or in combinations of them.Apparatus of the invention can be implemented using a computer programproduct tangibly embodied in a machine-readable storage device forexecution by a programmable processor; and method steps of the inventioncan be performed by a programmable processor executing a program ofinstructions to perform functions of the invention by operating on inputdata and generating output. The invention can be implementedadvantageously in one or more computer programs that are executable onprogrammable systems including at least one programmable processorcoupled to receive data and instructions from, and to transmit data andinstructions to, a data storage system, at least one input device, andat least one output device. Each computer program can be implemented ina high-level procedural or object-oriented programming language, or inassembly or machine language if desired; and in any case, the languagecan be a compiled or interpreted language. Suitable processors include,by way of example, both general and special purpose microprocessors.Generally, a processor will receive instructions and data from aread-only memory and/or a random access memory. Generally, a computerwill include one or more mass storage devices for storing data files;such devices include magnetic disks, such as internal hard disks andremovable disks; magneto-optical disks; and optical disks. Storagedevices suitable for tangibly embodying computer program instructionsand data include all forms of non-volatile memory, including by way ofexample semiconductor memory devices, such as EPROM, EEPROM, and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM disks. Any of the foregoing canbe supplemented by, or incorporated in, ASICs (application-specificintegrated circuits).

To provide for interaction with a user, the invention can be implementedon a computer system having a display device such as a monitor or LCDscreen for displaying information to the user. The user can provideinput to the computer system through various input devices such as akeyboard and a pointing device, such as a mouse, a trackball, amicrophone, a touch-sensitive display, a transducer card reader, amagnetic or paper tape reader, a tablet, a stylus, a voice orhandwriting recognizer, or any other well-known input device such as, ofcourse, other computers. The computer system can be programmed toprovide a graphical user interface through which computer programsinteract with users.

Finally, the processor can be coupled to a computer ortelecommunications network, for example, an Internet network, or anintranet network, using a network connection, through which theprocessor can receive information from the network, or might outputinformation to the network in the course of performing theabove-described method steps. Such information, which is oftenrepresented as a sequence of instructions to be executed using theprocessor, can be received from and output to the network, for example,in the form of a computer data signal embodied in a carrier wave. Theabove-described devices and materials will be familiar to those of skillin the computer hardware and software arts.

It should be noted that the present invention employs variouscomputer-implemented operations involving data stored in computersystems. These operations include, but are not limited to, thoserequiring physical manipulation of physical quantities. Usually, thoughnot necessarily, these quantities take the form of electrical ormagnetic signals capable of being stored, transferred, combined,compared, and otherwise manipulated. The operations described hereinthat form part of the invention are useful machine operations. Themanipulations performed are often referred to in terms, such as,producing, identifying, running, determining, comparing, executing,downloading, or detecting. It is sometimes convenient, principally forreasons of common usage, to refer to these electrical or magneticsignals as bits, values, elements, variables, characters, data, or thelike. It should remembered however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities.

The present invention also relates to devices, systems or apparatus forperforming the aforementioned operations. The system can be speciallyconstructed for the required purposes, or it can be a general-purposecomputer selectively activated or configured by a computer programstored in the computer. The processes presented above are not inherentlyrelated to any particular computer or other computing apparatus. Inparticular, various general-purpose computers can be used with programswritten in accordance with the teachings herein, or, alternatively, itcan be more convenient to construct a more specialized computer systemto perform the required operations.

A number of implementations of the invention have been described.Nevertheless, it will be understood that various modifications can bemade without departing from the spirit and scope of the invention.Accordingly, other embodiments are within the scope of the followingclaims.

8 EXEMPLARY USE—AN EPIDEMIC INVESTIGATION

To provide an example of how the invention might be used, the followinghypothetical situation has been created. The situation is described, theanalytic team is described, and then the process of analyzing thesituation using an exemplary embodiment of the current invention isdescribed. As will be apparent to those who have understood the abovedisclosure, the described exemplary embodiment is only one embodiment ofthe invention, and should in no way be seen as limiting on otherexemplary embodiments.

8.1 The Situation

A number of people have been falling ill, with some dying, in a limitedgeographic area near a military base that stores secret “specialmunitions” and where secret weapons development is done. The people whoare getting sick are all residents of a nearby town. There are alsomining operations in the area that have been in existence for severaldecades, with poorly supported accusations that the local ground watersupply has been affected. There has been a drought for the prior twoyears, following a five year period of abnormally high rainfall. Thishas resulted in an increase in the local rodent population, who are nowinvading human-inhabited areas looking for food.

The state health authorities have requested assistance from the Centersfor Disease Control (CDC), which has sent a medical investigation teamto work on the problem. The CDC team, due to the potential involvementof the military base research facility, has requested assistance fromthe Department of Defense (DoD), which has assigned some of its ownexperts, both medical and engineering, from the nearby base to assistwith the military's security aspects of the investigation. Due to thepotential for the event to be a terrorist attack rather than anaccident, the FBI has assigned an agent to monitor the investigationfrom Washington and report back if any indications of terrorism arediscovered.

8.2 Investigation Team Grouping

The assembled investigation team is divided into several groups, basedon security classifications, medical expertise, terrorism expertise, andexperience with structured analytic methods in this type ofinvestigation. Some analysts are members of more than one group. Groupmembership is used in several ways, both to advance the investigationand to maintain required security.

A “military” group is created for the DoD team members. Membership inthis group will be used to control access to information, where themilitary group will have access to relevant restricted militaryinformation, while the other groups will not. Only the DoD team memberswill be members of this group. All information elements that includerestricted military information or concepts are tagged as“military-restricted” and rules are created to limit viewing and use ofitems tagged “military-restricted” to members of the “military” group,so that they will be viewed and manipulated only by members of themilitary group.

A “medical” group is created for those with medical expertise. Medicalgroup membership will be used for adding weight to ratings by medicalgroup members, when the rating involves a medical issue. All medicalexperts (state, CDC, or DoD) will be members of the medical group.Hypotheses, relevant information, and indicators that require medicalexpertise to fully understand are tagged with a “medically-related” tag.This is used in the rules created for the purpose to grant extra weighton judgments relating to these items to members of the medical group.

A group named “CSI” is created for the FBI agent. CSI group membershipgrants permission to view various system outputs and participate indiscussions with other investigation team members, but does not grantpermission to enter any other inputs to any aspect of the system (MHG,ACH, QC, or IV). Should indications that terrorism or other criminalactivity is involved begin to surface, the permissions for this groupwill be changed to permit fuller participation, but until such time, theCSI group member is just an observer.

An “expert” group is created for those team members with successfulexperience with use of structured analytic techniques in this type ofinvestigation and using the system of the invention. Expert groupmembership is used to add weight to all ratings made by its members.

In addition to the military, medical, CSI and expert groups, there areother standard groups that are automatically generated by the system fora project, such as an “owner” group for the project owner(s), an “admin”group for those with permission to make changes to the configurationsettings for the project (such as defining or editing rules) or to groupmemberships, and an “ex-member” group for those members who have leftthe investigation team. Using the ex-member group to record departedteam members permits the departed member accounts to remain in place sothat discussion references, ratings, other group memberships, etc. madebefore departure remain valid and available to the remaining teammembers with the required permissions to view them (e.g. if a discussionentry is made in an ACH cell visible only to members of the militarygroup, members of the military group would continue to have access tothe discussion entry, but other team members who are not in the militarygroup would continue to see nothing, or to see an alternate entrydisplay, depending on how the system is configured. Discussion entriesmade in areas that are visible to all team members would continue to bevisible to all team members). Membership in the “ex-member” groupdisables all access and actions on the project. Should the member returnto the team, simple removal from the ex-member group returns them totheir prior status.

The owner of the project, i.e. the person leading the investigation orsomeone appointed by them, creates the project in the system, definesthe needed non-default groups, defines initial tags for use incharacterizing relevant information, hypotheses, and other informationelements, and sets up the rules used to specify privileges granted tothe defined groups, judgment weighting factors associated with definedgroups, filtering of displayed information, hypotheses, or otherinformation elements, thresholds for cut-off or clustering decisions,and other required project configuration settings and definitions.

8.3 Example Investigation Using the Invention

The first step for the team membership as a whole is to collect allrelevant information, and to tag it appropriately. Tagging is used oninformation elements to permit the automated analytic to referencetagged items as item classes for various purposes, such as filtering forview suppression, references in rules used to assign weightings,decisions about which QC matrices to assign to which groups, etc.Tagging of relevant information is usually done at the time theinformation is entered into the system, but an appropriately privilegedteam member, such as the project owner, can add or remove tags at anytime there is need to do so. In the example investigation we areconsidering, all restricted military information is tagged as“military-restricted”, and information that requires medical training tocomprehend properly is tagged “medically-related”. Where viewing ormanipulation of relevant information must be restricted to a specificgroup or groups, members of the group or groups perform the informationentry and tagging. When viewing and manipulation is permitted by allinvestigation team members, any analyst can enter and tag relevantinformation, unless there is a privilege restriction that prevents it.For example, the project owner could create a rule that permits relevantinformation to be entered only by the project owner, or only by a memberof an “admin” group, or by a special “information entry” group. Bydefining privileges and restrictions using groups and rules set up foreach project, great flexibility is made available for permitting orrestricting capabilities on a project by project basis. Each project canbe set up as its needs dictate.

After gathering and tagging what relevant information is alreadyavailable, the team needs to generate as many hypotheses for the causeof the illnesses as they can. They will then compare each hypothesisagainst the relevant information using the ACH automated analytic todetermine which hypotheses are most inconsistent with the relevantinformation and therefore unlikely to be valid. If a complete set ofhypotheses are generated, and all but one can be ruled out by beinginconsistent with relevant information, it is likely that the remaininghypothesis is correct. Using relevant information to eliminate all butone hypothesis, and confirming that the one remaining hypothesis iscorrect is the goal of the investigation. To generate an initial set ofhypotheses, the team decides to use the MHG automated analytic.

The MHG automated analytic requires a hypothesis, issue, activity, orbehavior to process. Typically a lead hypothesis is selected for this(one that it is felt by team members to be the most likely hypothesis).The team members each have some opinions as to what the cause of theillness might be, given the relevant information already known. Theymeet, in person and/or through the analyst discussion feature and sharetheir candidate hypotheses. After some discussion, the team decides toselect the hypothesis that there has been a leak of toxic or biologicalmaterials from the military base that is affecting those in thevicinity. The characteristics (e.g., the who, what, where, when, why,and how) of this hypothesis are then requested by the system, determinedby the analysts, and plausible alternatives determined and input intothe system:

-   -   Who is responsible for the leak: a researcher, a technician, an        unknown party.    -   What is leaking: a toxic substance, a biological . . . details        of possibilities are restricted to the military, and known only        by those in the military group.    -   Where is the source of the leak: the military base research        labs, the military base material storage area, a vehicle        delivering materials.    -   When did the leak occur: Over a long period of time, beginning        in the recent past, a one-time release.    -   Why did the leak occur: Accident, ignorance, experiment,        sabotage.    -   How is the leaking material reaching the victims: Through the        air, through the ground water, through personal contact, through        escaped lab animals.

All permutations of these alternatives are then generated by the system.For example, a researcher released a toxic substance at the researchlabs into the air over a long period of time by accident (such asthrough a piece of faulty equipment). A researcher released a toxicsubstance at the research labs into the air over a long period of timethrough ignorance (i.e. didn't realize it would persist long enough tocause harm). The number of permutations can be large, and automatedgeneration of all possible combinations is very efficient and greatlyspeeds up the process. The MHG automated analytic performs this taskautomatically, and presents the set of resulting permutations forevaluation.

Those generated hypotheses that involve restricted military informationare tagged as “military-restricted”, and visible only to those in themilitary group. In some exemplary embodiments, the system automaticallypropagates tagging from the alternatives used to construct a generatedhypothesis to the generated hypothesis so as to preservecompartmentalization of information. In some other exemplary alternativeembodiments, tagging is propagated manually by analysts. In yet otherexemplary alternative embodiments, tagging is propagated according todefined rules.

Once all hypotheses are generated, each team member rates eachhypothesis that is visible to them as to credibility on a zero to fivescale, where a zero means the hypothesis is illogical or makes no senseand should be discarded, and one to five refer to increasing levels ofcredibility. The credibility ratings are then averaged to calculate acredibility score. Those with a credibility score of zero, i.e. rated asillogical or not-sensible by all team members with permission to accessthem, are discarded. Discarded hypotheses are retained by the system foraudit purposes, but are not made available to the ACH automated analyticfor evaluation and play no further part in the analysis. The remaininghypotheses are sorted by credibility score, and a cutoff threshold isused to determine which hypotheses are most deserving of attention andthese are automatically loaded into the ACH automated analytic forevaluation against relevant information.

In the ACH automated analytic, the hypotheses generated with the MHGautomated analytic, as well as any others input by team members withpermission to add hypotheses, are matched against the currently knownrelevant information, and rated for consistency. The rating techniquecomprises determining whether each item of relevant information is veryconsistent with, consistent with, inconsistent with, very inconsistentwith, or neutral to each hypothesis.

As each analyst is rating relevant information against hypotheses intheir personal ACH matrix, they are shown only those hypotheses anditems of information that the compartmental restrictions permits them tosee and work with. What each team member is shown is based on the mostpermissive compartmental restrictions for the team member. For example,if relevant information is restricted to members of the military group,a member of the medical or CSI groups would be unable to view or workwith it, unless that person is also a member of the military group.Their membership in the medical or CSI group does not disqualify themfrom viewing the restricted information, but it does not qualify themeither. Only membership in the military group does that, under the rulesdefined for this project.

In some embodiments, compartment restrictions can restrict access to ananalyst's personal ACH matrix. For example, the restrictions may permitdisplay of the matrix, to allow discussion about cells in the matrix,but not allow rating cells or engaging in other activities. The CSIgroup members have this sort of permission configuration. This allowsthe FBI team member to follow the progress of the analysis, to see thehypotheses under consideration and to view the relevant information, andto participate in discussions about these, but not to affect the courseof the analysis directly by adding hypotheses, rating relevantinformation against them, or identifying assumptions or indicators.Compartment restriction is also used to restrict rating of medicalhypotheses or medically relevant information by non-medical teammembers, such as the DoD engineers, while permitting them to see thosehypotheses or information, or to make comments about them duringdiscussions.

Group membership can affect how a member's ratings are used whencalculating diagnosticity or when making other calculations. When ahypothesis, item of relevant information, or other item is tagged asbeing “medically-related”, members of the medical group receive anincreased weight for their ratings. Members of the “expert” groupreceive an adjustment for their ratings regardless of how the item istagged. A member of both medical and expert groups would have theirratings adjusted by both weights. The amount of adjustment, and whetherit increases or decreases a calculated value, is determined by the rulesdefined in the project configuration, which is set by someone in the“owner” group. Members of the owner group also configure which group orgroups the weighting applies to. Not all groups effect weighting. Forexample, membership in the military group conveys no weighting factor.

When the group matrix is displayed, the content is limited tohypotheses, relevant information, combined ratings, discussions, etc.that are viewable by all team members, unless a team member withpermission to do so specifically requests additional information beincluded. When making such a request, the team member can specify whichadditional group memberships should be used to determine what toinclude. The available options for group memberships will include onlythose possessed by the requesting team member. For example, if relevantinformation element A is tagged as military-restricted, it will not bedisplayed unless a team member who is a member of the military grouprequests it. If a hypothesis is restricted to either military or expertgroup members, and a team member who is a member of both groups ismaking a request to display additional information, the team member canspecify that display be based on either group membership. Such requeststo override default displays are logged, and can require a specificacknowledgement of intent (i.e. “Please confirm override of securityrestriction on display of military-restricted information”, with arequirement to enter an authentication to prove group membership in themilitary group before the information is displayed).

When a hypothesis or item of relevant information is suppressed ineither the group matrix or in a personal matrix, it is replaced by analternate version. The alternate version indicates that the informationelement is being suppressed, and why. For example, “Hypothesis requiresmilitary group membership for viewing”, or “Item of restricted militaryinformation viewable only by military group members”. In someembodiments, an alternate description can be specified for restrictedentries when viewed by those not possessing membership in a requiredgroup. For example, “Military hypothesis alpha”, or “Accidental spill oftoxic chemical”, rather than the more specific hypothesis descriptionthat would be shown to someone in the required group, such as“Accidental release of substance X-148 from building 12 on or aboutSeptember 12”. The text of the alternate description is in red toindicate that the actual description is being suppressed.

Once ratings have been applied, diagnosticity calculated, and hypothesessorted, selected hypotheses can be made available to the QC automatedanalytic for use in generating additional hypotheses. Selected ones ofthe generated hypotheses can then be returned to the ACH automatedanalytic for evaluation against relevant information to see which areconsistent with known information and which are not.

Where there is insufficient relevant information with high enoughdiagnosticity value, indicators can be specified and made available tothe IV automated analytic where they will be rated for diagnosticity andsorted into a priority ordering. Selected indicators can also beinvestigated or monitored to generate additional relevant informationfor inclusion in the ACH matrix.

To increase the chance that all valid hypotheses are being considered,team members select hypotheses from the ACH matrix and send them to theQC automated analytic. The QC automated analytic generates additionalhypotheses by breaking a selected hypothesis into its componentassumptions, generating contrary assumptions for each assumption, andthen putting pairs of contrary assumptions into two-by-two matrices inall possible combinations. Team members then concoct at least oneplausible story for each quadrant of each two-by-two matrix, and thenidentify indicators for each resulting hypothesis.

When the initial hypothesis made available to the QC automated analyticis restricted as to which team members can see it, only those teammembers who participate in the QC automated analytic may participate inrating the matrix. For example, if the hypothesis chosen is that therewas an accidental release of substance X-148 from building 12 on orabout September 12, only military group team members participate. If thehypothesis is not restricted, such as a hypothesis that it is anaturally occurring illness being spread by rodents that happen to livein the tribal lands, all team members can participate.

Even when all team members can participate, there can be reasons forlimiting participation to a subset of team members. For example, toshorten the total time to process all of the top hypotheses through theQC automated analytic, the team can break into smaller sub-teams and dothem in parallel. Or if understanding a particular hypothesis involvesspecialized knowledge, a group made up of those with the most expertisein that area can deal with that hypothesis. In this example case, themilitary group members deal with the hypotheses that are restricted totheir group, while the other team members deal with the unrestrictedhypotheses.

Since some of the hypotheses deal with sabotage, which could beterrorism-related, adjustments are made to the group permissions toallow the CSI group member to participate, so that the team can have thebenefit of FBI input into the formation of contrary assumptions andstory creation.

The resulting stories are re-formulated as valid hypotheses and sentback to the ACH automated analytic for evaluation against relevantinformation, while any indicators generated for the hypotheses are madeavailable to the IV automated analytic for validation and prioritizing.

Indicators, whether from the ACH or QC automated analytics, or thoseinput by appropriately authorized team members, need to be evaluated tomake sure that they are diagnostic, and prioritized so that limitedresources are used in the most effective manner. This is done using theIV automated analytic.

Indicators and hypotheses are automatically arranged in a matrix similarto that used for the ACH automated analytic, hypotheses in columns andindicators on the rows, and are then assessed for the likelihood thateach indicator would occur in the associated hypothesis. When analystsrate indicators, the order of presentation can be different for eachteam member, using the survey techniques described above. Team membersassign likelihood ratings to each cell in the matrix using the HL, L, C,U, or HU ratings of the IV automated analytic. These are used by the IVautomated analytic to calculate a diagnosticity rating for eachindicator. Indicators with diagnosticity ratings below a specifiedthreshold are displayed “grayed out” to indicate that they are out ofconsideration for the hypotheses being considered. These non-diagnosticindicators are not simply deleted, but are retained in an inactive stateso that team members will be reminded that they have already beenconsidered. Also, should additional hypotheses be added in future, theindicators' diagnosticity rating could change and make them valid.

As with other parts of the system, those hypotheses, indicators and theassociated ratings that are restricted to being viewed by specificgroups within the team are visible only to those team members in thosegroups. For example, an indicator consisting of a test for the presenceof material X-148 would be visible only to military group members, andevaluated only by them. Likewise, any hypotheses involving materialX-148 would also be limited to military group member viewing. Members ofother groups would, depending on configuration settings, either seenothing, or see only a substitute display, such as “restrictedhypotheses #n”, or “release of a military-restricted material”. In somecases it can be advantageous to permit members of groups that arerestricted from viewing full details of an indicator or hypothesis tonevertheless rate the indicator. This is done using the alternatedescriptions. For example, the hypothesis might be shown to therestricted team members as “release of military-restricted substance #1from the base”, and an indicator shown as “detection ofmilitary-restricted substance #1”. It is not necessary to know thenature of the substance to know that detection of the substance would bea highly likely indication of the hypotheses involving its release.

Also as with other parts of the system, the ratings of individual teammembers can be weighted, based on group membership, such that some teammembers have greater effect on the final indicator diagnosticity ratingsthan others. For example, the configuration rules can be set such thatthose in the “expert” group have their ratings count twice, or theirindividual diagnosticity ratings can be multiplied by a weighting factorbefore the group consensus value is calculated.

Once indicators have had their diagnosticity calculated, and those withlow diagnosticity marked, the indicators are sorted into hypothesisorder based on their home hypotheses, and then by diagnosticity. Ifthere are hypotheses with an insufficient number of valid indicators,the team members will develop additional indicators and the process willbe repeated for the added indicators. Otherwise, indicators areprioritized by various factors including diagnosticity, cost, likelihoodof deception, difficulty of obtaining valid information, etc., and thetop indicators selected for monitoring. As monitoring of indicatorsgenerates relevant information, it is added to the ACH system matrix andused to re-evaluate hypotheses.

1. A computer-implemented method for providing compartmented,collaborative, integrated, automated analytics to analysts, comprising:selecting a computer-encoded contextual workflow; determining acomputer-encoded compartment manager, said computer-encoded compartmentmanager including computer-encoded information about the context of saidcontextual workflow; retrieving said computer-encoded information aboutthe context; selecting a computer-implemented automated analytic usingsaid computer-encoded contextual workflow; providing under control ofsaid computer-encoded compartment manager said information about thecontext to said automated analytic; processing said computer-encodedinformation using said computer-implemented automated analytic, togenerate thereby analytical information representing an outcome to saidanalysts; and processing said analytical information in accordance withsaid computer-encoded compartment manager and said computer-encodedcontextual workflow.
 2. The computer-implemented method of claim 1,wherein said contextual workflow includes at least one contextualattribute selected from the group consisting of: guidance to theautomated analytics, as to the process to be followed, information touse as inputs, information required for outputs, and any requiredlabeling, tagging, and compartmentalization.
 3. The computer-implementedmethod of claim 2, wherein guidance to said automated analytics furtherincludes guidance for analysts.
 4. The computer-implemented method ofclaim 1, wherein said contextual workflow defines rules based upon oneor more aspects of said context.
 5. The computer-implemented method ofclaim 4, wherein said contextual workflow defines rules for eachanalyst, each project, for each installation of the system, or by thesystem design.
 6. The computer-implemented method of claim 1, whereinsaid computer-encoded context manager executes under computer control atleast one function selected from the group consisting of: generating orassigning tags associated with specific information elements, or withspecific types of information elements within a compartment; generatingor assigning compartments associated with specific information elements,or with specific types of information elements within a compartment;managing requests to, and information elements provided by, a data storeto enforce rules for information access, tagging, and association rules;assigning or associating information elements or types of informationelements with specific tags, associations, controls, contexts, orcompartments; assigning or associating rules with information elementsor types of information elements that require specific tagging orrestrictions to be applied to newly created information elements andrestricting the availability of information elements or types ofinformation elements to which a requestor is not authorized access oruse.
 7. The computer-implemented method of claim 1, wherein saidcomputer-encoded context manager executes under computer control atleast one function selected from the group consisting of: implementingaccess controls over information elements; implementing controls overtagging and association among multiple information elements; andenforcing information segregation of information elements, includinglogical and physical segregation of information elements among differentdata stores.
 8. The computer-implemented method of claim 1, furthercomprising providing a set of rules defining the scope of visibility ofinformation, said rules being effective to define private information,restricted information, and unrestricted information.
 9. Acomputer-implemented system for providing compartmented, collaborative,integrated, automated analytics to analysts, said system comprising: acomputer-controlled service configured to select a computer-encodedcontextual workflow; a computer-controlled service configured todetermine a computer-encoded compartment manager, said computer-encodedcompartment manager including computer-encoded information about thecontext of said contextual workflow; a computer-controlled serviceconfigured to retrieve said computer-encoded information about thecontext; a computer-controlled service configured to selectcomputer-implemented automated analytic using said computer-encodedcontextual workflow; a computer-controlled service configured to provideunder control of said computer-encoded compartment manager saidinformation about the context to said automated analytic; acomputer-controlled service configured to process said computer-encodedinformation using said computer-implemented automated analytic, togenerate thereby analytical information representing an outcome to saidanalysts; and a computer-controlled service configured to process saidanalytical information in accordance with said computer-encodedcompartment manager and said computer-encoded contextual workflow. 10.The computer-implemented system of claim 9, wherein said contextualworkflow includes at least one contextual attribute selected from thegroup consisting of: guidance to the automated analytics, as to theprocess to be followed, information to use as inputs, informationrequired for outputs, and any required labeling, tagging, andcompartmentalization.
 11. The computer-implemented system of claim 10,wherein guidance to said automated analytics further includes guidancefor analysts.
 12. The computer-implemented system of claim 9, whereinsaid contextual workflow defines rules based upon one or more aspects ofsaid context.
 13. The computer-implemented system of claim 12, whereinsaid contextual workflow defines rules for each analyst, each project,for each installation of the system, or by the system design.
 14. Thecomputer-implemented system of claim 9, wherein said computer-encodedcontext manager executes under computer control at least one functionselected from the group consisting of: generating or assigning tagsassociated with specific information elements, or with specific types ofinformation elements within a compartment; generating or assigningcompartments associated with specific information elements, or withspecific types of information elements within a compartment; managingrequests to, and information elements provided by, a data store toenforce rules for information access, tagging, and association rules;assigning or associating information elements or types of informationelements with specific tags, associations, controls, contexts, orcompartments; assigning or associating rules with information elementsor types of information elements that require specific tagging orrestrictions to be applied to newly created information elements andrestricting the availability of information elements or types ofinformation elements to which a requestor is not authorized access oruse.
 15. The computer-implemented system of claim 9, wherein saidcomputer-encoded context manager executes under computer control atleast one function selected from the group consisting of: implementingaccess controls over information elements; implementing controls overtagging and association among multiple information elements; andenforcing information segregation of information elements, includinglogical and physical segregation of information elements among differentdata stores.
 16. The computer-implemented system of claim 9, furthercomprising providing a set of rules defining the scope of visibility ofinformation, said rules being effective to define private information,restricted information, and unrestricted information.
 17. Acomputer-readable medium containing computer-readable program controldevices thereon, said computer-readable program control devices beingconfigured to enable a computer to provide compartmented, collaborative,integrated, automated analytics to analysts by causing said computer toexecute computer-controlled operations comprising: selecting acomputer-encoded contextual workflow; determining a computer-encodedcompartment manager, said computer-encoded compartment manager includingcomputer-encoded information about the context of said contextualworkflow; retrieving said computer-encoded information about thecontext; selecting a computer-implemented automated analytic using saidcomputer-encoded contextual workflow; providing under control of saidcomputer-encoded compartment manager said information about the contextto said automated analytic; processing said computer-encoded informationusing said computer-implemented automated analytic, to generate therebyanalytical information representing an outcome to said analysts; andprocessing said analytical information in accordance with saidcomputer-encoded compartment manager and said computer-encodedcontextual workflow.
 18. The computer-readable medium of claim 17,wherein said contextual workflow includes at least one contextualattribute selected from the group consisting of: guidance to theautomated analytics, as to the process to be followed, information touse as inputs, information required for outputs, and any requiredlabeling, tagging, and compartmentalization.
 19. The computer-readablemedium of claim 18, wherein guidance to said automated analytics furtherincludes guidance for analysts.
 20. The computer-readable medium ofclaim 17, wherein said contextual workflow defines rules based upon oneor more aspects of said context.
 21. The computer-readable medium ofclaim 20, wherein said contextual workflow defines rules for eachanalyst, each project, for each installation of the system, or by thesystem design.
 22. The computer-readable medium of claim 17, whereinsaid computer-encoded context manager executes under computer control atleast one function selected from the group consisting of: generating orassigning tags associated with specific information elements, or withspecific types of information elements within a compartment; generatingor assigning compartments associated with specific information elements,or with specific types of information elements within a compartment;managing requests to, and information elements provided by, a data storeto enforce rules for information access, tagging, and association rules;assigning or associating information elements or types of informationelements with specific tags, associations, controls, contexts, orcompartments; assigning or associating rules with information elementsor types of information elements that require specific tagging orrestrictions to be applied to newly created information elements andrestricting the availability of information elements or types ofinformation elements to which a requestor is not authorized access oruse.
 23. The computer-readable medium of claim 17, wherein saidcomputer-encoded context manager executes under computer control atleast one function selected from the group consisting of: implementingaccess controls over information elements; implementing controls overtagging and association among multiple information elements; andenforcing information segregation of information elements, includinglogical and physical segregation of information elements among differentdata stores.
 24. The computer-readable medium of claim 17, furthercomprising providing a set of rules defining the scope of visibility ofinformation, said rules being effective to define private information,restricted information, and unrestricted information.